Interview Question: "Describe a challending Project you worked on"

 

Describe a Challenging Project You Worked On

One common interview question is to "Describe a challenging project you worked on." In 2011, AWS had only a few RDS choices, unlike the many options available today.

Throughout my career, I have worked on many interesting projects, but I will focus on my time at Precor, a fitness equipment manufacturer. In 2011, Precor was building on the concept of "Connected Fitness" to allow fitness machines to connect to the internet. This enabled users to download workouts, save workouts, watch instructional videos, read e-books while running on a treadmill, and enjoy many other features.

Precor needed to build a team from the ground up for this project, as they previously only made fitness machines with basic embedded software. I was the second hire after the engineering manager and became Principal Engineer. My mission was to design and build on the vision of "connected fitness." Precor had two main teams: one working on the console (P80 equipment, a dedicated terminal attached to a fitness machine) and my team, working on the backend systems powering all machines in gyms, clubs, and hotels.

Team Composition:

My team consisted of an Engineering Manager, a Principal Engineer (me) as team lead/architect, five engineers, and a Product Owner. We followed a Scrum approach.
 

My Mission:

I was tasked in building two categories of APIs (myself and the team, I played a key role, leading the team, in both defining much of the architecture and code modules):

1. APIs to help club operators understand machine utilization.  
2. APIs to empower exercisers.

Focus on the Exerciser's API:

Every console connected to the internet had a dedicated UI running on an embedded Linux machine, powered by an ARM Cortex CPU with decent video capability, such as rendering YouTube videos. The Fitness Equipment (FE) served as an API client. Another client type was the Mobile App, built by a third party.

The backend systems were built as microservices running on the AWS Cloud. The Exerciser API was a REST API leveraging OAuth2 for user authorization. The use case for exercisers was to create and track their fitness goals using both a mobile app and the fitness machines, regardless of location, as long as they were using Precor connected fitness equipment.

For club owners, the use case was to better serve their customers with modern machines and understand machine utilization, idle time, and receive custom alerts for machine malfunctions. They could also generate custom reports on user exercise frequency to predict membership cancellations.

Exerciser API Features:

The Exerciser API allowed users to log in via RFID, enabling the machine to adjust settings such as angle, inclination, and speed on a treadmill, and start recording exercise data, including calories and duration. Users could check their daily and weekly exercise progress towards their goals on the mobile app, which could be customized for goals like getting fit or losing weight. Users were awarded badges on the mobile app for achieving milestones, such as 1,000 steps, accompanied by a congratulatory message and a cool image.

The Exerciser API:

 

 

On the backend, we built the stack with Java and the Spring framework, using Apache as the HTTP server. The database was RDS (MySQL), and we used DynamoDB, a columnar datastore, for high volume and write throughput. Redis was used to track denial of service attacks.

DynamoDB was utilized for the Fitness Machine APIs to store frequent heartbeat data and log messages. To buffer between the server and storage, we implemented a message queue in front of DynamoDB.

The Goals of this Project, known as Preva was to Increase user retention, attract new members, 

drive secondary revenue generation and help gym members achieve their goals.

Precor conducted several studies to measure each of these goals, and I will link these studies to Precor website:

•Increase retention

•Attract new Members

•Drive Secondary Revenue Generation

•Help Gym Members Achieve their Goals

Conclusion:

This project was both fun and challenging due to several factors: tight deadlines, the innovation involved, learning cloud computing, leading a team, and ultimately helping people improve their lives by promoting a healthy lifestyle.

Spring Boot Camper and REST Assured Testing Library

I almost never blog, but I will attempt once more. Hopefully I will make it a habit.

I have recently created a sample project to demonstrate how simple and cool it is to write REST APIs using Spring Boot. Also, how to test it with integration tests that runs as fast as unit tests using REST Assured.

This is going to be a series of samples, each focused in showcasing one aspect of the framework or technique.
The first one is  camp_rest_assured,  which demonstrate how to integrate Spring Boot with REST Assured library, with an extra bonus of showing a technique that I created using a custom converter service to do request validation using custom messages collected from representation beans annotated with JSR-303 validations.

Checkout the sampler project here:
https://github.com/phavelar/boot-camper

I will be adding detailed explanations soon. Stay Tuned!

Grails URL Encoding

Bug: Form data being encoded twice in Grails 2.1.1

I recently run into a bug related to character encoding in Grails 2.1.1 application that manifested a priory only when the web app was deployed into production.  

All tests where passing during CI builds as well as local development environment.

It was a tricky issue to figure it out, so I want to share it here, hopping to save you some time if you encounter this problem and are fortunate enough to find this post :)  

Say you're dealing with i18n and have UTF-8 form encoded text data (application/x-www-form-urlencoded), somehow after posting a text in Cyrillic, the contents were being doubly encoded, messing up the original text data..

The issue does not manifest in development mode,  or rather, I discovered that if you use IntelliJ IDE to launch the web application ("exploded mode" )  all is normal, that is the Cyrillic text is properly encoded.  Instead,  if you build the war using "grails war" command and manually deploy it to Tomcat, then this bug happens.

Digging deeper we had the following sloppy code to encode the form data:

URLEncoder.encode(formData)

As you can see from the Java Doc API,  this is a deprecated method, where the resulting string may vary depending on the platform's default.

I'm not sure why the platform default changes when packaging the war via "grails war" command versus running the war from within the IDE, but the fact is that this had cost us a few hours spent on debugging.

Method Summary
static String	encode(String s)           Deprecated. The resulting string may vary depending on the platform's default encoding. Instead, use the encode(String,String) method to specify the encoding.
static String	encode(String s, String enc)           Translates a string into application/x-www-form-urlencoded format using a specific encoding scheme.

To fix, simply change the code to URLEncoder.encode(formData, "UTF-8")

So, discovering the behavior about packaged versus exploded war phenomena  was half the battle to be able to reproduce this bug.   However, this could have been avoided altogether if the developer had paid attention to compiler deprecated warnings. Or  not done this:

-Dgrails.log.deprecated=false  //to turn off for development mode  

Hope this post can help some fellow developers !

Happy Coding !

 

Using Guava Ranges to Implement Rules-based Logic

Consider the requirements of implementing an awarding mechanism based on goal completion percentage.
This kind of requirements is commonly found in loyalty programs. For example, if the user achieves between 70% to 79% of  his/her original goal, the user gets  a 10 points reward. Likewise, if the user completes 80% to 89%  the user gets 15 points and so on.  The basic idea is to have a set of percentage ranges associated with points. In addition, the percentage ranges/points combination must changeable to support evolving requirements.
I'm currently working on a Fitness Application, so I'm going to use this domain to illustrate a coding technique for Rules-based logic without using explicit IF statements.  In this fitness application, we wish to award "badges" to an exerciser, based on the percentage of goal completion. 

The first thing that comes to mind, is to implement a series of  "if-else" blocks to determine the percentage range the exerciser is at according his goal completion: 

 int lookupPoints(int goal)   
 {  
   int points = 0;    
   if (goal >= 70 && goal <= 79){  
     points = 10;  
   }   
   else if (goal >=80 && goal <= 89){   
     points = 15;  
   }  
   else if (goal >=90 && goal <= 99){  
     points = 25;  
   }   
   else if (goal >=100 && goal <= 109){  
     points = 50;  
   }  
   else if (goal > 110){  
     points = 60; 
   }  
   return points;  
 }  

This approach may be fine, but it is spaghetti code.  As requirements change you may  have to add or modify the existing range boundaries and point values.

Another approach to consider is leveraging a rules engine. I think using a rules engine, like Drools is overkill for this simple case.

Is there is a way to accomplish the same thing, without using any explicit "if"s while making possible to easily modify the ranges and point values?
Yes, after all you're reading this to find out how !  With this blog technique, there is no need to change the underlying decision making algorithm in case requirements change, all you need to do is to change the data setup, just like a  fixture.

What kind of sorcery is this ?  Guava  libraries to the rescue!

Enter the Range class. (follow the link for it's Java Doc)
A range (or "interval") defines the boundaries around a contiguous span of values of some Comparable type. So we can express all intervals in the code block above this way:

 Ranges.closedOpen(new Integer(70, new Integer(79));  
 Ranges.closedOpen(new Integer(80, new Integer(89));  
 Ranges.closedOpen(new Integer(90, new Integer(99));  
 Ranges.closedOpen(new Integer(100, new Integer(109));
 Ranges.atLeast(new Integer(110));   

What we need next is a way to associate each range with its "points" value.
We can use a Map for that like that:

 Map<Range<Integer>, Integer> pointAwardMap = new HashMap<Range<Integer>, Integer>();  
 pointAwardMap.put(Ranges.closedOpen(new Integer(80, new Integer(89)), 10);  
 pointAwardMap.put(Ranges.closedOpen(new Integer(90, new Integer(99)), 15);   
 // ...  etc, until   
 pointAwardMap.put(Ranges.atLeat(new Integer(110), 60);  

Now that we have the point associated with a range, we need a way to do the look up based on the current goal value.  That can be accomplish using the Guava Collections2  Filter class:

 int lookupAwardPoints(final Integer completedGoal)  
 {      
    Collection<Range<Integer>> percentileRange = filter(pointAwardMap.keySet(), new  Predicate<Range<Integer>>()  
    {  
      @Override  
      public boolean apply(Range<Integer> input)  
      {  
        return input.contains(completedGoal);  
      }  
    });  
    return percentileRange.isEmpty() ? 0 : pointAwardMap.get(percentileRange.iterator().next());  
 }  

So, now that you know the though process, all that is left is to organize the code in a nice clean way, removing duplication and making it easy to add new rules:

1:  import com.google.common.base.Predicate;  
2:  import com.google.common.collect.Range;  
3:  import com.google.common.collect.Ranges;  
4:  import java.util.Collection;  
5:  import java.util.HashMap;  
6:  import java.util.Map;  
7:  import static com.google.common.collect.Collections2.filter;  
8:  public class PointAward  
9:  {  
10:    {  
11:      addRangeAndPointAward(70, 80, 10);  
12:      addRangeAndPointAward(80, 90, 15);  
13:      addRangeAndPointAward(90, 100, 25);  
14:      addRangeAndPointAward(100, 110, 50);  
15:      addRangeAndPointAward(110, 60);  
16:    }  
17:    private Map<Range<Integer>, Integer> pointAwardMap = new HashMap<Range<Integer>, Integer>();  
18:    public int lookupAwardPoints(final Integer percentCompleted)  
19:    {  
20:      Collection<Range<Integer>> percentileRange = filter(pointAwardMap.keySet(), new Predicate<Range<Integer>>()  
21:      {  
22:        @Override  
23:        public boolean apply(Range<Integer> input)  
24:        {  
25:          return input.contains(percentCompleted);  
26:        }  
27:      });  
28:      return percentileRange.isEmpty() ? 0 : pointAwardMap.get(percentileRange.iterator().next());  
29:    }  
30:    private void addRangeAndPointAward(int lowerEnd, int pointAward)  
31:    {  
32:      pointAwardMap.put(Ranges.atLeast(lowerEnd), pointAward);  
33:    }  
34:    private void addRangeAndPointAward(int lowerEnd, int upperEnd, int pointAward)  
35:    {  
36:      pointAwardMap.put(Ranges.closedOpen(lowerEnd, upperEnd), pointAward);  
37:    }  
38:  }  

Note that all you need to do if your rules change is to modify the "fixture" like code on lines 10-15,  no need to ever change the looupAwardPoints()  function.  Also notice that we don't have any ifs, of spaghetti code.
Granted this code is way more sophisticated and complex than the first one, but it give you a lot of flexibility.

I hope, I showed you a useful trick!

Until next time!

A custom JSR-303 compliant validator for apache CommonsMultipartFile

Recently on my Spring 3 MVC project, I needed to implement file upload/download capability - nothing too exciting here, but I wanted to use JSR-303 and have my controller to cleanly check for a valid file type and size (my project only allows for uploading PDF files with 30Mb max), that is to move that constraint logic away from the controller into a JSR-303 validator. Since I'm a nice guy, I'm sharing the code here :) here is an excerpt of my controller code:

1:  import org.slf4j.Logger;  
2:  import org.springframework.beans.factory.annotation.Autowired;  
3:  import org.springframework.security.core.context.SecurityContextHolder;  
4:  import org.springframework.stereotype.Controller;  
5:  import org.springframework.ui.Model;  
6:  import org.springframework.validation.BindingResult;  
7:  import org.springframework.web.bind.annotation.ExceptionHandler;  
8:  import org.springframework.web.bind.annotation.RequestMapping;  
9:  import org.springframework.web.bind.annotation.RequestMethod;  
10:  import javax.validation.Valid;  
11:  import static org.slf4j.LoggerFactory.getLogger; 
12:  @Controller  
13:  @RequestMapping(value = "/upload")  
14:  public class UploadController  
15:  {  
16:    private static final Logger LOGGER = getLogger(UploadController.class);  
17:    @Autowired  
18:    private ProjectManagementService projectManagementService;  
19:    @RequestMapping(method = RequestMethod.GET)  
20:    public String getUploadForm(Model model)  
21:    {  
22:      String viewName = "upload/uploadForm";  
23:      User user = (User) SecurityContextHolder.getContext().getAuthentication().getPrincipal();  
24:      if (user.getAuthorities().contains(Authority.ROLE_ADMIN))  
25:      {  
26:        viewName = "redirect:admin/adminAccessList";  
27:      }  
28:      else if (user.getAuthorities().contains(Authority.ROLE_USER))  
29:      {  
30:        model.addAttribute(new ProjectUploadCommand());  
31:      }  
32:      else  
33:      {  
34:        throw new IllegalStateException("Unauthorized access - should never happen.");  
35:      }  
36:      return viewName;  
37:    }  
38:    @RequestMapping(method = RequestMethod.POST)  
39:    public String upload(@Valid ProjectUploadCommand projectUpload, BindingResult result)  
40:    {  
41:      String viewName = "upload/uploadForm";  
42:      if (!result.hasErrors())  
43:      {  
44:        final User user = (User) SecurityContextHolder.getContext().getAuthentication().getPrincipal();  
45:        Project project = new Project(user,  
46:            projectUpload.getProjectIdeaTitle(),  
47:            projectUpload.getDescription(),  
48:            projectUpload.getEstimatedLaunchDate(),  
49:            new ProjectContent(projectUpload.getFileData().getOriginalFilename(), projectUpload.getFileData().getBytes()));  
50:        projectManagementService.addProject(user, project);  
51:        viewName = "upload/thankYou";  
52:      }  
53:      return viewName;  
54:    }  
55:    @ExceptionHandler  
56:    public String catchAllHandler(Exception ex)  
57:    {  
58:      LOGGER.error("Unexpected Exception at UploadController", ex);  
59:      return "errorPage";  
60:    }  
61:  }  

Note the @Valid annotation in front of ProjectUploadCommand object. (I know, a better name for it would be ProjectUploadForm since its not a real command in the GOF sense, but its how some Spring folks call these objects) here is the "command" object: CommonsMultipartFileValid.java

1:   import java.lang.annotation.Documented;  
2:   import java.lang.annotation.Retention;  
3:   import java.lang.annotation.RetentionPolicy;  
4:   import java.lang.annotation.Target;  
5:   import javax.validation.Constraint;  
6:   import javax.validation.Payload;  
7:   import static java.lang.annotation.ElementType.ANNOTATION_TYPE;  
8:   import static java.lang.annotation.ElementType.CONSTRUCTOR;  
9:   import static java.lang.annotation.ElementType.FIELD;  
10:  import static java.lang.annotation.ElementType.METHOD;  
11:  import static java.lang.annotation.ElementType.PARAMETER;

  
12:   @SuppressWarnings({"UnusedDeclaration"})  
13:   @Target({ANNOTATION_TYPE, METHOD, FIELD, PARAMETER, CONSTRUCTOR})  
14:   @Retention(RetentionPolicy.RUNTIME)  
15:   @Constraint(validatedBy = CommonsMultipartFileValidator.class)  
16:   @Documented  
17:   public @interface CommonsMultipartFileValid  
18:   {  
19:     String fileSize() default "31457280";  
20:     String[] supportedFileTypes() default {"pdf"};  
21:     String message() default "File must be a PDF no larger then 30M bytes";  
22:     Class[] groups() default {};  
23:     Class[] payload() default {};  
24:   }  

and here is the implementation: (left a TO DO for you !) CommonsMultipartFileValidator.java

1:   import java.util.Arrays;  
2:   import java.util.List;  
3:   import javax.validation.ConstraintValidator;  
4:   import javax.validation.ConstraintValidatorContext;  
5:   import org.springframework.web.multipart.commons.CommonsMultipartFile;  
6:   /**  
7:    * A custom JSR-303 validator for CommonsMultipartFile objects.  
8:    * validates the file extension and file size.  
9:    *  
10:   * @author Paulo Avelar  
11:   */  
12:   public class CommonsMultipartFileValidator implements ConstraintValidator<commonsmultipartfilevalid commonsmultipartfile="commonsmultipartfile">  
13:   {  
14:     private String fileSize = "31457280";    //30M-bytes  
15:     private String[] supportedFileTypes = {"PDF"};  
16:     @Override  
17:     public void initialize(CommonsMultipartFileValid constraintAnnotation)  
18:     {  
19:       fileSize = constraintAnnotation.fileSize();  
20:       supportedFileTypes = constraintAnnotation.supportedFileTypes();  
21:     }  
22:     @Override  
23:     public boolean isValid(CommonsMultipartFile value, ConstraintValidatorContext context)  
24:     {  
25:       boolean result = false;  
26:       try  
27:       {  
28:         if (value != null)  
29:         {  
30:           result = validateFileSize(value);  
31:           //two step process because MIME type identification can be costly  
32:           if (result)  
33:           {  
34:             result = validateExtensionType(value);  
35:           }  
36:         }  
37:       }  
38:       catch (Exception e)  
39:       {  
40:         throw new RuntimeException(e);  
41:       }  
42:       return result;  
43:     }  
44:     //  TODO: use mime type identifier instead of silly extension name  
45:     private boolean validateExtensionType(CommonsMultipartFile value)  
46:     {  
47:       int dotPos = value.getOriginalFilename().lastIndexOf(".");  
48:       boolean result = false;  
49:       if (dotPos != -1)  
50:       {  
51:         String extension = value.getOriginalFilename().substring(dotPos + 1);  
52:         final List<string> supportedExtensions = Arrays.asList(supportedFileTypes);  
53:         for (String supportedExtension : supportedExtensions)  
54:         {  
55:           if (extension.equalsIgnoreCase(supportedExtension))  
56:           {  
57:             result = true;  
58:             break;  
59:           }  
60:         }  
61:       }  
62:       return result;  
63:     }  
64:     private boolean validateFileSize(CommonsMultipartFile value)  
65:     {  
66:       boolean result = false;  
67:       if (value != null)  
68:       {  
69:         if (value.getSize() != 0 &amp;&amp; value.getSize() &lt;= Long.valueOf(fileSize))  
70:         {  
71:           result = true;  
72:         }  
73:       }  
74:       return result;  
75:     }  
76:    }  

Almost forgot, here is the Unit Test for it: CommonsMultipartValidatorTest.java

1:   import java.io.File;  
2:   import java.io.IOException;  
3:   import java.io.InputStream;  
4:   import java.io.OutputStream;  
5:   import java.io.UnsupportedEncodingException;  
6:   import java.lang.annotation.Annotation;  
7:   import javax.validation.Payload;  
8:   import org.apache.commons.fileupload.FileItem;  
9:   import org.hamcrest.Matchers;  
10:   import org.junit.Test;  
11:   import org.springframework.web.multipart.commons.CommonsMultipartFile;  
12:   import static org.hamcrest.MatcherAssert.assertThat; 

 
13:   public class CommonsMultipartValidatorTest  
14:   {  
15:     private CommonsMultipartFileValid commonsMultipartFileValid = new CommonsMultipartFileValid()  
16:     {  
17:       @Override  
18:       public String fileSize()  
19:       {  
20:         return "10000";  
21:       }  
22:       @Override  
23:       public String[] supportedFileTypes()  
24:       {  
25:         return new String[]{"PDF", "DOC"};  
26:       }  
27:       @Override  
28:       public String message()  
29:       {  
30:         return "whatever message";  
31:       }  
32:       @Override  
33:       public Class[] groups()  
34:       {  
35:         return null;  
36:       }  
37:       @Override  
38:       public Class[] payload()  
39:       {  
40:         return null;  
41:       }  
42:       @Override  
43:       public Class annotationType()  
44:       {  
45:         return null;  
46:       }      
47:     };      
48:     @Test  
49:     public void validateCommonsMultipartValidator()  
50:     {  
51:       CommonsMultipartFileValidator validator = new CommonsMultipartFileValidator();  
52:       validator.initialize(commonsMultipartFileValid);  
53:       assertThat(validator.isValid(new CommonsMultipartFile(new ValidFileItem()), null), Matchers.is(true));  
54:       assertThat(validator.isValid(new CommonsMultipartFile(new InvalidFileItem()), null), Matchers.is(false));  
55:       assertThat(validator.isValid(new CommonsMultipartFile(new InvalidFileItemSize()), null), Matchers.is(false));  
56:     }  
57:     private class ValidFileItem implements FileItem  
58:     {  
59:       @Override  
60:       public InputStream getInputStream() throws IOException  
61:       {  
62:         return null;  
63:       }  
64:       @Override  
65:       public String getContentType()  
66:       {  
67:         return null;  
68:       }  
69:       @Override  
70:       public String getName()  
71:       {  
72:         return "test.pdf";  
73:       }  
74:       @Override  
75:       public boolean isInMemory()  
76:       {  
77:         return false;  
78:       }  
79:       @Override  
80:       public long getSize()  
81:       {  
82:         return 450;  
83:       }  
84:       @Override  
85:       public byte[] get()  
86:       {  
87:         return new byte[0];  
88:       }  
89:       @Override  
90:       public String getString(String encoding) throws UnsupportedEncodingException  
91:       {  
92:         return null;  
93:       }  
94:       @Override  
95:       public String getString()  
96:       {  
97:         return null;  
98:       }  
99:       @Override  
100:       public void write(File file) throws Exception  
101:       {  
102:       }  
103:       @Override  
104:       public void delete()  
105:       {  
106:       }  
107:       @Override  
108:       public String getFieldName()  
109:       {  
110:         return null;  
111:       }  
112:       @Override  
113:       public void setFieldName(String name)  
114:       {  
115:       }  
116:       @Override  
117:       public boolean isFormField()  
118:       {  
119:         return false;  
120:       }  
121:       @Override  
122:       public void setFormField(boolean state)  
123:       {  
124:       }  
125:       @Override  
126:       public OutputStream getOutputStream() throws IOException  
127:       {  
128:         return null;  
129:       }  
130:     }  
131:     private class InvalidFileItem extends ValidFileItem  
132:     {  
133:       @Override  
134:       public String getName()  
135:       {  
136:         return "test.XXX";  
137:       }  
138:     }  
139:     private class InvalidFileItemSize extends ValidFileItem  
140:     {  
141:       @Override  
142:       public long getSize()  
143:       {  
144:         return 9999999;  
145:       }  
146:     }  
147:   }