Google BERT in Recognizing Police Brutality
Human Rights First is a nonprofit organization that is concerned with human rights on an international level. For 40 years they have worked hard to bring to light the struggle and fight for human rights. By demanding reform accountability and justice, they make an impact for those whose rights have been threatened and abused.
For the past four weeks I have had the privilege of working with a great team of Data Scientists and Web Developers to contribute to one such project. Blue Witness is a product that Human Rights First is developing to identify and track cases of Police brutality. They would hold Officers accountable that choose to abuse their power and seek justice for those that have been wronged and for their families.
Blue Witness
Starting out, one of my concerns was: How do we make sure to get this right? This was an enormous task we were taking on, and it came with a lot of social impact.
At the time, the product functioned by scraping tweets from the Twitter API and running them through a K-NN model. The model then placed each tweet into a rating bucket depending on the type of Police involvement and action that was taken. If a tweet was considered to be a True case, it was Ranked and sent to the Administrator dashboard for approval. This was where they investigate the incident to find if it really was a true case of Police brutality.
We found that the K-NN model, at its previous state, was not very accurate and needed some refinement. Also, the words that describe the “Ranks” for the model to parse through could have been better.
Google BERT
When we had our first stakeholder meeting, we had decided that we wanted to implement a Google BERT model in order to accurately predict true cases of Police brutality. A teammate and I decided that we would take on the task to create a Google BERT model for this binary classification problem. The feedback that we received for this project, was that in order for the stakeholder to see value in BERT, we had to be able to show a custom prediction. This meant making it possible to pass in any string and having the model return a prediction.
BERT stands for Bidirectional Encoder Representations from Transformers. It is a transformer-based machine learning model that is used for Natural Language Processing and is what Google utilizes for Google Translate and their Search Engine.
With BERT, you can use one of their pre-trained models and fine-tune it to your specific task. So, we were able to fine-tune BERT to be able to predict true instances of Police brutality parsed from tweets sourced from the Twitter API.
I do not want to go too into the ins and outs of BERT. I will provide a link to a video that a teammate and I recorded showcasing the code we used for Google BERT and what the goals are moving forward with the model.
The Challenge
Data Sourcing was the biggest obstacle for this project. In order for BERT to successfully and accurately predict true instances of Police brutality from a tweet, it has to know what that looks like. The data that we trained and tested BERT on was far from perfect. We really just wanted to get it up and running.
So, this will be a task for the next group of engineers taking on the project. After creating a dataset that contains true cases of Police brutality, there is likely to be a shortage of tweets compared the tweets that the model will be trained on as not cases of Police brutality. This could require balancing the training set using Synthetic Data. I believe this will be something the next group should consider when moving forward with BERT.
The next part of the project that needs to be refined is the Ranking system. After BERT finds the True cases, they will be input into a K-NN model to be parsed into the Ranking categories. There should be a way to use Natural Language Processing to find the words that are the most used in tweets in each rank so as to categorize them correctly. This would be more relevant and accurate than thinking of words for each category on your own.
During the last stakeholder meeting, they also discussed doing this in reverse. (Using the output of a K-NN model as the input to BERT.) This is a whole other issue that will require careful and detailed planning.
I believe the most important feature of this project is the Administrator’s investigation. This is what ultimately decides if it is a true case and needs to be acted upon. This is what will also make the model more and more accurate. As the Admin identifies true and false cases, these should be fed back into the model in order to improve performance and accuracy. Over time, I believe the model could be extremely accurate and the Admin would see less and less false cases.
Here’s to the Future
Tomorrow our team is leaving the project and moving on to new things. I am excited and yet, sad to say good-bye. It has been an amazing experience to get to work with such a talented group of people and on such an important and meaningful product. I hope to be able to further use my skills to make a difference in the world. I just need to get my foot in the door.
Here’s to what the future holds! Cheers!
