My First Internship project!

Building a microservice iteratively

My second internship project: https://grokwithrahul.medium.com/my-second-internship-project-e72f8f7865c9

I’m a 15-year-old in the 9th grade, and I’m delighted to have joined FirstRoot as an Intern.

FirstRoot is a Silicon valley startup that aims to improve financial literacy in children and teenagers, through the usage of participatory budgeting. Students learn by investing real money in schools, schools benefit by seeing a real lasting impact, and the community wins when these money-smart children become adults. Every Participatory Budgeting Cycle (PB Cycle) has a series of phases that the students go through, from planning to refinement to voting to implementation.

Let’s consider a PB cycle Admin persona. This Administrator is in charge of a Participatory Budgeting Cycle (or PB Cycle) which has been launched in a school. At FirstRoot, we know that this persona needs up-to-date information on important statistics, which includes student participation, proposal costs, and voting results. When we started out, the team didn’t worry too much about this aspect: we wanted to focus our resources on improving the student experience. After all, there are a few hundred students to each admin.

As a startup, we do our best to leverage free services to get things done. Google docs are a robust set of free services. FirstRoot started with a pseudo-manual approach, running a few python scripts on behalf of teachers when they requested a data report, which pulled data from our back-end database, formatted it, and wrote it to a Google Sheet shared with the Administrator.

We are an agile organization that aims to constantly improve it’s offering to customers on a weekly, sometimes daily basis. After a while, this task had started to take way too much time and resources to execute. As a logical next step, we needed to extend this prototype into a full blown microservice which automated everything, from start to finish.

Creating this microservice was my first job as an intern. With a lot of mentoring and guidance from our Chief of Product, Clint Gossett, I’m super excited to see my first microservice being deployed into production.

How was this service built?

This Microservice is written in Python. I used Heroku to host our service — Heroku manages the scaling and building, and it integrates with GitHub to automatically deploy any new commits. The web framework used is Flask, running with the Web Server Gunicorn. The update/creation of a single PB Cycle Sheet is classified as a job, and these jobs are broken down into tasks. Using a message broker (Redis) and a task queue (Celery), I implemented a task-based architecture that enables the microservice to continue to service other tasks, if one fails.

The benefit of this architecture is that workers can spawn nearly instantly to handle the excess load during periods of high traffic.

Getting Data from Backend

Instead of using the more traditional REST API framework, we use GraphQL for our API query language. GraphQL has a singular endpoint from which users can access all data. The Schema and type system are documented. The service sends a query to the GraphQL server, which includes Authorization codes and data requirements. The GraphQL server responds with a JSON object. This object contains the data which matches the requirements.

The service sends a Query for a list of all the active PB Cycles, and extracts their IDs, and then sends a Query for every PB Cycles to retrieve all data surrounding it.

As each PB Cycle Sheet gets updated, its data gets temporarily stored in a Pandas dataframe, while it is in transit from the Backend to the Sheets.

Creating/Updating Google Sheets

The service pulls the corresponding Google Sheet ID using the Google Drive API and edits it using the Google Sheets API. If a corresponding sheet for the PB Cycle does not exist, then the service creates one using a template.

Bug Reporting/Logging

I used Bugsnag to automatically catch any bugs and send mail detailing information about the problem. Furthermore, the service maintains precise logs of every request and task initiated in the system. A logger instance is copied from component to component to allow each component to capture information itself.

Status Management

Obviously, users want information on their request — has it finished? Were there any issues? For one-off requests to update a single PB Cycle, there isn’t much time needed to service the request then return the status within a few seconds. However, it takes a lot of time (a few minutes) to service requests asking for the update of all PB Cycles. By then, the connection with the user would have died. Therefore, the service maintains a ticketing system where each of these requests is given an id and a slot in a database, and the status corresponding to each id keeps getting updated as the request gets serviced. The users can see the status of their request by Querying the ticketing system with the ID it had provided them.

Testing

There is nearly complete code coverage, by unit tests. Mocks were used to isolate components and test them individually. This testing framework is integrated into our release cycle by Github’s CI/CD tool, which runs these tests over every commit.

Conclusion

Compared to the previous method used to share data with the Admin persona, this service cuts the time and effort needed to manually create a sheet with all columns and update it by running scripts for each PB Cycle. The Admin persona is also convenienced by reliable, frequent information updates on the PB Cycle they are managing.

Personally, this was an awesome first project! A thousand lines of code later, I learnt the importance of test-driven and the need for complete unit tests. I learnt about building and maintaining a proper microservice. I learnt how to work with people all around the globe! I learnt that I still have a lot to learn! Thank you, everyone from FirstRoot, for helping me understand daily standups, code demos, and feedback sessions — what a fun team! A special thanks to the Head of Product Clint Gossett, who patiently spent a lot of time helping and encouraging me. FirstRoot’s CEO, Luke Hohmann, is an active member and long-time contributor of the global agile movement. I am learning about iterative and agile software development from Luke and the team, about not building a block of software and shipping it to customers, but iteratively learning from customers to piece together the larger solution. Looking forward to new challenges to help FirstRoot succeed and our users achieve financial literacy!