Google vs Meta,
A Turf War I Caused

What I learned from publishing controversial benchmarks

January 2025

About a year ago, a hot debate took place on X.com involving the chief scientist of Google and the chief AI scientist of Meta. Unfortunately, I was the engineer behind the scene who conducted the controversial experiments causing the incident. To avoid getting involved in the fight, I had to remove almost all my information from the internet and had to block people on GitHub for unfriendly behaviors.

Now, it no longer attracts any public attention. It is finally a good time to review what happened and put down the valuable learnings I got from this rare experience.

DISCLAIMER

The article represents the writer's personal opinion and in no way reflects the opinions and/or ideas of Google or its partners.

The entirety of the information presented within this article is sourced exclusively from publicly available materials.

To maintain focus and avoid potential misinterpretations, the experimental results, while publicly accessible, have been intentionally omitted from this article.

Furthermore, as I have no direct knowledge of the internal processes undertaken by Google in response to the incident, these processes are outside the scope of this discussion.

What happened?

About a year ago, I benchmarked some popular deep learning frameworks for their running speed as part of my work at Google. The results favored TensorFlow and JAX, developed by Google, over PyTorch, developed by Meta. I carefully wrote a blog post with multiple rounds of revision and feedback from my teammates before posting it on our official website and making the code available online. One of our teammates also authored a thread on X to promote it. That was where the fight started.

People started to quote the results table out of context and post it on X, which caused an argument between some very influential people from Google and Meta, including the chief research scientists from both sides. One of the main developers of PyTorch from Meta responded on X by redoing the experiments and writing a list of flaws in the experiments I had done. This post really escalated the issue and drew huge attention in the community.

One day after that post, all influential people involved in this incident removed their related posts from X with a follow-up statement saying that they had reached a peaceful agreement not to benchmark with each other's framework without notifying each other beforehand.

What did I do?

I really did not do much when this thing took off on X. There was nothing much I could do as an individual contributor with no influence in the community.

I decided to focus on GitHub only because I had to since it was mainly my responsibility. So, I kept interacting with people on GitHub, fixing the problems in my code, and updating the posted results.

I followed these principles as I did my job.

• Get the attitude right. I benchmark the frameworks to benefit the community. I do this to keep them informed about the state-of-the-art performance, not to show the superiority of one over another.

• Acknowledge the problems in my code and be grateful. Fix them ASAP, and be grateful to the people who pointed them out. They helped me gain a lot of new knowledge about deep learning frameworks and systems.

• Always be friendly to people. Never get into an argument with anyone.

• Block people, not their opinions. For the people who are not so friendly, block them, remove their comments, and replace them with a summary of their main points. Removing their opinions would look like I was hiding something from the public.

• Stay invisible on the internet. To mitigate the risks the incident posed on my career, I stayed away from it as much as possible unless it was my responsibility. I removed all my information from the internet, including everything on X and LinkedIn. I really believe in the dark forest theory of the web. Those who expose themselves on the internet will inevitably be attacked by either bots or humans.

What did I learn?

The following are the valuable things I learned from this rare experience.

Benchmarks are controversial.

All benchmarks on others' work are controversial. I tested multiple frameworks, each of which has a bunch of stakeholders. As long as there is a gap in the results, some of the stakeholders might be unhappy about it. So, anticipate the criticism when publishing the results.

Be extra objective.

Due to the huge amount of attention the benchmark attracted, people would exaggerate and criticize any unfair setting or unintentional mistakes. So, we need to be extra objective about the experiments and not preoccupied with any idea leaning towards any side.

People quote out of context.

The long-form blog posts or academic papers are just too long for people to read, especially in the era of the attention economy when short-form content is preferred and promoted by various platforms. People will cut out the most eye-catching part of the long-form post, for example, the results table comparing different frameworks, and share it on social media.

Examine what could be quoted out of context as a countermeasure before posting it online. Include as much context as possible in the tables and figures so that, when quoted, the context is quoted with them.

Personal impact vs corporate interest

A great piece of advice from my PhD advisor at our graduation was: Don't focus on publications anymore. This advice was for the people who were joining the industry instead of academia. I took it literally without understanding what it really meant until recently. After a couple of years of working in the industry, I finally discovered something much more profound in this advice. It highlights the fundamental difference between working in academia and the industry.

Academia cares about personal impact. Publishing research papers is an effective method to achieve that goal. There are other methods, but they all contribute to the same ultimate goal: building personal impact.

The industry cares about profit. No matter how famous you are as a researcher or an engineer, you are evaluated based on your direct or indirect contributions to the company's bottom line.

These two goals may not always align with each other. The incident I caused is a great example. Fighting with each other online is more about making a personal impact. You get more exposure in the community and claim your technical superiority if you win. However, it could also waste corporate resources and potentially damage the company's public image. It is not a wise thing to do from a business perspective. That was why both sides decided to remove all the posts in the end.

If I really want to thrive in the industry, though personal impact could be beneficial, I should always go for corporate interest when there is a conflict between the two. Even if I want a larger personal impact, it is no longer purely about my expertise, professional skills, or research credits. I should build a professional image that understands the bigger picture and does the right thing for the organization.

Returning to the advice of not focusing on publications, what it really says is that do not mindlessly aim for personal impact but align it with the corporate interest instead.

The benefits of reduced online presence

As I mentioned above, I removed all my information from the internet. So, now, no one can easily see my degrees, job level, or my past achievements. There are some unexpected benefits of staying invisible.

When something about me is publicly visible, I want to polish it. When my job level was displayed on my LinkedIn profile, I wanted to get to the next level as soon as possible. I also had the pressure of regularly pulling out new shiny achievements to share on social media.

These pressures could benefit my career, but not always. Focusing on things with short-term returns could distract me from my long-term career goal. Reducing my online presence allows me to rethink and focus on what is really important to me. For example, I spend more time on my essential skills, including English communication, reading, and writing. I build real connections with people rather than expand a random online audience.

My impact is next level.

When I publish papers during my PhD, I had to advertise it as much as possible. No one really cared about them. Now, I quietly did the work without advertising it at all, but it caused a big fight online. My impact is next level now. I should be more cautious about what I say and what I do.

I am still very junior.

Throughout the incident, from the start of the argument on X to the removal of all the posts, there definitely were a lot of internal discussions at Google and external communications between Google and Meta on this issue. None of them involved me in the process, even though I was the engineer who conducted all the experiments. I was only asked to remove the controversial results from the post when they made the final decision.

So, I am still junior, doing assigned tasks, not making decisions. No one mentioned me when they were fighting on X. It is a big relief for me. I am junior enough to focus on my work without too many worries. It will be the big names taking the damage, not me.

Google has an amazing culture.

Google has an amazing corporate culture. I can easily imagine that, in a company with a bad culture, the engineer who actually did the job would be blamed when there was a problem. However, as I mentioned, I was only asked to remove the results in the end. No one blamed me at all.

Final words

This rare experience helped me learn a lot, especially the attitude of dealing with the incident and the mindset shift from academia to the industry. Though it could be a little stressful as I navigate through this, I feel super grateful for the valuable learnings it gave me.