Yesterday, I started a short series of posts about my conclusions from a year and a half of professional academic research. I outlined what I believe is a growing crisis in science. Today, want to talk about what we can do about it. My point today is this:

The world desperately needs amateur scientists.

Because of what’s happened to science in the mainstream, science that’s outside of it is more important than ever. While doing science professionally, I was astonished at how little time I spent pursuing new ideas compared to what I was used to as an amateur. This wasn’t down to any kind of error in direction or a problem with the mentorship I received. Quite the reverse. The lack of innovation was down to unavoidable elements of the culture of academia. The mentoring I received helped me navigate that culture. Without it, I would have been lost.

What I spent most of my time doing instead of innovating was checking my work to make sure that it was perfect, and that all possible bases were covered in order to make sure the work was of publishable quality. I spent months of my time trying to compensate for a huge number of hypothetical yes-buts that people might employ to try to invalidate my research. Scientists do this because in the world they inhabit, they need to. Anything less than this is a recipe for career-fail. Getting something wrong makes you look bad.

However, in a very important abstract sense, this is the wrong way to do science. That’s because no one can ever adequately check their own work. That’s what the community should be for. But a point has been reached in most areas of science where there are huge career incentives to make it look as if each piece of research that you deliver is correct, even though this goal is unattainable.

As Karl Popper pointed out years ago, science advances through refutation. You can never know what’s true, only what’s false. Thus, the most effective way for science to move forward is for people to make bold, falsifiable conjectures.

What this means in practical terms is that kudos in science should be attached to the act of checking someone else’s ideas, and to the creation of falsifiable conjectures, not their correctness. In short, science would be far better off to adopt a hard-earned lesson from the tech industry: embrace failure.  Wear each failure as a badge of pride. 

However, because the scientific culture we have puts a premium on ownership of an intellectual landscape, and on apparently uncontestable expertise, we deliver science in units of apparent advancement instead.

This is slowly creating a nightmare of confirmation bias in today’s scientific community. With the addition of computer simulations and powerful statistical tools, the point of finding yourself looking at what you want to see has become extremely easy to attain. All the repetitive checking and testing does nothing whatsoever to alleviate this, because the tests that people do are, by and large, those that occur to them after the process of bias has already occurred.

This is not to say that all science done today is wrong. Far from it. Plenty of good work gets done. But that is despite of the system that we have set up, rather than because of it. Science is so hard to do as a career precisely because we’ve gone and made it that way.

There are those who will tell you that science has to look the way that it currently does. They will tell you that, because there is so much research being done, it is the responsibility of scientists to check their own work as thoroughly as possible before publishing it. This stance however, misses the point.

Certainly, keeping up to date on publications (such as Arxiv preprints) in many scientific fields these days can be exhausting. But this is because A: it creates an inferiority complex in young scientists struggling to compete, and B: because the work that’s published is usually so cautious and glutinous in its presentation that reading even one paper can take hours, let alone the dozen that might be posted on a given day. I have yet to meet a scientist of my generation who reads professional science papers for fun.

In a better world, the papers would be written like tutorials, and students would scour Arxiv every day as a goldmine of refutation opportunities. However,  even with all the best will in the world, it will take years, or some exceedingly painful crisis, or both, for the world of professional research to change. To my mind, then, the role of the amateur scientist in today’s world is to do what professionals can seldom afford to do any more: play.

If human knowledge is going to advance fast enough for us to solve the problems we create for ourselves, there need to be people who can try on offbeat ideas without fear. Amateur scientists need to be deliberately operating outside of the mainstream, and ignoring prestige.

This is a key point. Via my work on digital physics, I’ve spoken to a lot of smart, well-intentioned people with interesting ideas who are what people in the physics community often refer to as ‘crackpots’. What this really means is that they’re untrained people who’re giving an idea they care about their best shot. The problem that most of them share is that they’re trying urgently to communicate their reasoning to professionals.

Usually, the crackpot/amateur-scientist narrative goes like this:

I think I’ve found something really important. It looks to me like this could be the basis for a Theory of Everything. Now I just need to talk to a physicist to work out the math and communicate it to the world.

Physicists usually hate this because:

1: They have spent the last n years of their life trying to absorb a highly complex and nuanced way of looking at the world. When someone comes along talking a completely different language, it’s somewhere between annoying and outright painful to then adapt to comprehend someone’s new esoteric frame of reference, particularly when they start off with no proof that there is something concrete behind it.

2: They have so little time to push forward on their own work that they’re desperately keen to pursue their own ideas, not someone else’s. This is because the culture they inhabit tells them that this is what counts, and will either make or break their already fragile career.

3: They’re fighting for every inch of their credibility. The last thing they need is to try attaching themselves to someone with no credibility, and without any formal training in their field. For the professional scientist, it almost never looks like there’s anything to gain.

What this means is that lucky crackpot/amateurs in digital physics receive disdain when they reach out to physicists. The unlucky ones receive a sequence of uncomfortable pseudo-supportive brush-offs that send mixed signals and can last for years, crushing hope and seeding bewilderment.

But look at what causes this problem. These amateurs are chasing what they think science is about: sudden flashes of insight that transform how we think about the world and which reveal to the world the genius of their creator. Amateurs are chasing this dream because they see the professionals do it. The irony is that the professionals should know better.

The answer to this social malaise is simple. Amateur scientists should seek out each other instead and engage in peer review, rather than expecting professionals to help. This is better for everyone. And it makes sense, because, outside of critical input that could equally well be delivered by other amateurs, the best that a professional can offer in support is their credibility and prestige. And frankly, outside of the academic job market, these commodities aren’t that useful. An amateur’s ideas are surely at their strongest when there are enough people already excited about them that professional attention becomes inevitable.

What I’d like to see is a body of amateur scientists helping each other, refuting each other’s work, where possible, and using the internet as their greatest tool and forum. We are in a better position to do this now than at any point in human history. With services like Coursera providing a free scientific and technical education, and the web providing countless platforms to be heard, we are ready for a new amateur science renaissance.

But realistically how, you may ask? Surely scientific literacy takes years. Isn’t the frontier of human knowledge a huge distance away from most people’s understanding of how the world works? Hasn’t all the easy stuff already been exhaustively explored? Shouldn’t we just leave it up to the professionals?

Hell no. The edge of human knowledge is next door and it’s being very inadequately searched. And everyone who can cut code, think critically, and who cares, should be helping to fix that. In the next post, I’ll say more about how.