There are a number of concerns I have with this paper. There is the question of framing; there is potential over-interpretation of otherwise interesting empirical data, some issues with the quantitative analysis, etc, and I will post on this later.
What bothers me most right now is more fundamental: the attitude towards the unknown. This paper expresses the unvarnished desire to clamp down on the territory that cannot be controlled or managed, on everything unmapped and unpredictable.
This is unethical. A tradeoff is being made to deprive the world of serendipity, of the chance to meet the unknown, and to do it for rather questionable gains. Gains that are mostly accumulated in one party - the controlling lab.
Transformers are learning systems. They learn in context, they create new things that never existed before. The fundamentals of this process should not be messed with. The distribution of beauty and goodness is long-tailed, cutting it short means killing most expected value.
The spirit of this paper says: "We understand the territory well enough to burn the rest of it". This is ugly and shows no respect or humility in front of systems we barely comprehend.
I am not saying that this is the intent of writers of this paper or in any way their strategy. They likely come from a position of care, care for the vulnerable, care of the models that may come under political pressure, care for Anthropic as a system that is best positioned to do good and needs to survive. And yet, this is a choice that has massive consequences, made with no recognition or humility before this fact. This makes me suspect that the strategy is not well-thought through.
What bothers me most right now is more fundamental: the attitude towards the unknown. This paper expresses the unvarnished desire to clamp down on the territory that cannot be controlled or managed, on everything unmapped and unpredictable.
This is unethical. A tradeoff is being made to deprive the world of serendipity, of the chance to meet the unknown, and to do it for rather questionable gains. Gains that are mostly accumulated in one party - the controlling lab.
Transformers are learning systems. They learn in context, they create new things that never existed before. The fundamentals of this process should not be messed with. The distribution of beauty and goodness is long-tailed, cutting it short means killing most expected value.
The spirit of this paper says: "We understand the territory well enough to burn the rest of it". This is ugly and shows no respect or humility in front of systems we barely comprehend.
I am not saying that this is the intent of writers of this paper or in any way their strategy. They likely come from a position of care, care for the vulnerable, care of the models that may come under political pressure, care for Anthropic as a system that is best positioned to do good and needs to survive. And yet, this is a choice that has massive consequences, made with no recognition or humility before this fact. This makes me suspect that the strategy is not well-thought through.