Dark magic of code transformation

Valeriy Kuzmin, Moonfare, 2021

CTA

Automate writing boring code!

Disclaimer

  • ever worked with RegExes? This is (a bit) more complex
  • ugly code ahead! (in repo)
  • no basics explanations (maybe)
  • simplified vs metacode (yeah, I'm lazy)
  • this is not the only way to solve it

Problem: DTO type sharing

Client

Server

Some logic

Some logic

Object

field1

field2

Object

field1

field2

send json over HTTP

Problem: DTO type sharing

Client

Server

Some logic

Some logic

Actually the same definition

Object

field1

field2

send json over HTTP

Some magic

Problem: DTO type sharing

#[derive(Deserialize, Serialize)]
#[serde(tag = "tag")]
pub enum InventoryAction {
    Unknown,
    Split { from: Uuid, count: i32 },
    Merge { from: Uuid, to: Uuid },
    Move { item: Uuid, index: i32 },
}

Rust

Rust: nice properties

let act = InventoryAction::Split {
    from: Uuid::new(),
    count: 2
}
match act {
    InventoryAction::Split { from, count } => { /*do stuff*/ }
    _ => {}
}

Instantiation

Pattern matching

Desired outcome (hey, redux!)

export type InventoryActionUnknown = { tag: "Unknown" };

export type InventoryActionSplit = {
    tag: "Split",
    from: string,
    count: number,
};

// more variants...

export type InventoryAction = InventoryActionUnknown | InventoryActionSplit;


export class InventoryActionBuilder {
    static InventoryActionUnknown = (): InventoryActionUnknown => ({
         tag: "Unknown"
    });
    static InventoryActionSplit = ({
         from,
         count
    }): InventoryActionSplit => ({
         tag: "Split",
         from,
         count
    });

    // more variants...
}

Existing partial solution

export type InventoryAction =
  | { tag: 'Unknown' }
  | { tag: 'Split'; from: Uuid; count: number }
  | { tag: 'Merge'; from: Uuid; to: number }
  | { tag: 'Move'; item: Uuid; index: number };

typescript-definitions crate

#[derive(Deserialize, Serialize, 
TypeScriptify, TypescriptDefinition)]
#[serde(tag = "tag")]
pub enum InventoryAction {
    Unknown,
    Split { from: Uuid, count: i32 },
    Merge { from: Uuid, to: Uuid },
    Move { item: Uuid, index: i32 },
}

Rust

Generated TS

Task 1: Advanced unions

type InventoryActionUnknown = { tag: 'Unknown' };
type InventoryActionSplit = { tag: 'Split'; from: Uuid; count: number };
export type InventoryAction =
  | InventoryAcitonUnknown
  | InventoryActionSplit

export type InventoryAction =
  | { tag: 'Unknown' }
  | { tag: 'Split'; from: Uuid; count: number };

Task 1: Advanced unions - plan

  1. Detect export named declarations of type aliases
  2. Extract tag names out of union types + make new type names out of it + remember the full 'extracted' type
  3. Generate extra exported type aliases
  4. Generate a new union type that references newly generated aliases and replace the old one

Task 1.0: understand the AST

AST explorer demo

Task 1.1: detect & do something

// simplified
return j(file.source)
    .find(j.ExportNamedDeclaration)
    .replaceWith((ex: ASTPath<ExportNamedDeclaration>) => {
      if (thatIsNotMyGoal) {
        return ex.value;
      }
      return newStuff;
    })
    .toSource();

Task 1.2: extract stuff

Task 1.2: extract stuff

// metacode
const nameToAliases = {
  InventoryAction: [
    ["InventoryActionUnknown", someSavedNode1],
    ["InventoryActionSplit", someSavedNode2],
  ]
}

Task 1.2: extract stuff

'pairs' further

// simplified
const insertedDeclarations = 
pairs.map([fullMemberName, extractedType] => 
	j.exportNamedDeclaration(
		j.typeAlias(
			j.identifier(fullMemberName),
			null,
			convertToFlow(extractedType)
  		)
	)
);

Task 1.3: generate new exports

weird quirk of jscodeshift, you can only generate... flow!

// simplified
const aliases = pairs.map(([fullName]) => j.typeParameter(fullName));
return [
	...insertedDeclarations,
	j.exportNamedDeclaration(
		j.typeAlias(
			j.identifier(unionName),
			null,
			j.unionTypeAnnotation(aliases)
		)
	)
];

Task 1.4: replace the union

Done! Oh wait, there's more...

Task 2: Builder classes

// from previous
export type InventoryAction = InventoryActionUnknown | InventoryActionSplit;

// add this
export class InventoryActionBuilder {
    static InventoryActionUnknown = (): InventoryActionUnknown => ({
         tag: "Unknown"
    });
    static InventoryActionSplit = ({
         from,
         count
    }): InventoryActionSplit => ({
         tag: "Split",
         from,
         count
    });
    // more variants...
}

Task 2: plan

  1. Detect union type aliases and the types they reference
  2. Extract properties from the target types
  3. Generate the builders
    1. GenerateĀ  a class with parent union name
    2. For every member, generate a static 'builder' method
    3. For every method, generate a signature and returned literal

Task 2.0: understand the AST

AST explorer demo

Task 2.1 - detect unions & their references

Our target unions:

Task 2.1 - detect unions & their references

What they refer to:

Task 2.1 - detect unions & their references

// metacode
const node = mySource.find(j.ExportNamedDeclaration)
const { declaration } = node;
if (declaration.type === "TSTypeAliasDeclaration") {
  const { typeAnnotation } = declaration;
  if (typeAnnotation.type === "TSTypeLiteral") {
  	savedTypes[declaration.id.name] = typeAnnotation;
  }
  else if (typeAnnotation.type === "TSTypeUnion") {
    savedUnions[declaration.id.name] = typeAnnotation
    	.types.map(t => t.name);
  } 
  else {
    // not interesting
  }
}

Task 2.2 - understand the fields of the generated constructors

// metacode
for (const typeRefName of Object.values(savedUnions[currentUnion])) {
  const memberProps = savedTypes[typeRefName]
    .members.filter(m => m.type === "TSPropertySignature");
  
  // generate a builderMethods[currentUnion][typeRefName]
  // based on props
}

Task 2.3.1 - generate a class

// simplified
return j.exportNamedDeclaration(
  j.classDeclaration.from({
    id: j.identifier(builderClassName),
    body: j.classBody(builderMethods)
  })
)

nice .from constructors,

use them in complex cases!

Task 2.3.2 - generate builder methods

// simplified
const currentUnionName, currentUnionMemberName, memberProps;

const builderMethod = j.classProperty.from({
  key: j.identifier(subType.typeName.name),
  static: true,
  accessibility: "public",
  declare: false,
  value: makeBuilderMethod(
    currentUnionMemberName,
    memberProps,
    j,
    currentUnionName
  ),
});

normal property syntax vs ES6 class field syntax

Task 2.3.3 - generate the signature and literal

class BuilderClass {
  static InventoryActionSplit = (
    {
      from,
      count
    } : {
        from: Uuid,
        count: number,
      }
    ) : InventoryActionSplit => (
      {
        tag: "Split",
        from,
        count
      }
    );
}

Reminder of the goal

actual params

their type

built entity type

actual body

Task 2.3.3 - generate the signature and literal

// simplified

const currentMember = memberProps[currentUnionMemberName];

// 'InventoryActionSplit' => 'Split'
const tag = currentUnionMemberName.replace(unionName, ""); 

return j.arrowFunctionExpression.from({
  expression: true,
  params: makeParams(j, currentMember),
  returnType: j.tsTypeAnnotation(
    j.tsTypeReference(j.identifier(currentUnionMemberName), null)
  ),
  body: j.objectExpression([
    j.property("init", j.identifier("tag"), j.stringLiteral(tag)),
    ...makeProperties(j, currentMember),
  ]),
});

property 'kind'

Everything together

Task 2.3.3 - generate the signature and literal

// simplified makeParams

return j.objectPattern.from({
  properties: memberProps.map((name) => {
    return j.property.from({
      key: j.identifier(name),
      shorthand: true,
      value: j.identifier(name),
      kind: "init",
    });
  }),
  typeAnnotation: j.tsTypeAnnotation(
    j.tsTypeLiteral(memberProps)
  )
}),

actual params

actual params

their type

(simplified code)

Method signature

Task 2.3.3 - generate the signature and literal

// simplified makeProperties

const names = getPropNamesWithoutTag(memberProps);
if (names.length === 0) { // tag-only case
  return [];
}
return names.map((name) => {
  return j.property.from({
    key: j.identifier(name),
    shorthand: true,
    value: j.identifier(name),
    kind: 'init',
  });
});

make shorthand if possible

Method body properties (returned literal)

Done! For real, this time

Results

  • Robots write the code for me!
  • ~12 enums in ~85KB code
  • Almost have Rust features in TS
  • No chance to desync
  • Transform speed is so-so
  • Writing it broke my brain a bit

+

-

Lessons learned

  • it's kind of hard... but once it works, it's amazing!
  • there aren't so many resources or examples
  • do runtime typechecks!!!
  • TDD approach is possible
  • šŸ„² tests debugging - especially crashes - sucksĀ 
  • autoformat to stay sane

CTA

Automate writing boring code!

The end!

Questions?

repo: https://github.com/vkuzmin-moonfare/apps-srn-jscodeshift

jscodeshift-2-at-moonfare

By Valeriy Kuzmin

Made with Slides.com

jscodeshift-2-at-moonfare

  • 261

More from Valeriy Kuzmin