How do I use a regular transformer within a program transformer?

[LukasMachetanz]

Hey!

I am wondering of it is possible to get the TransformationContext within a Transformer Program (https://github.com/nonara/ts-patch#transforming-program)?

Thanks. Greets.

[nonara]

Effectively, using a "program transformer" requires a good understanding of the TypeScript Compiler API.

To help on that journey, I've created a short guide on how to get better acclimated with it. Please have a read here: Advice for working with the TS Compiler API

While I believe that will be helpful for you and others moving forward, I recognize that having a more complete example on how to do this would also be helpful, so I've written the following example:

Example of transforming a Progam with additional AST transformations

import ts, { CompilerHost, CompilerOptions, Program, SyntaxKind, TransformationContext, SourceFile, Node } from 'typescript';
import { PluginConfig, ProgramTransformerExtras } from "ts-patch";
import path from 'path';
import {} from 'ts-expose-internals'


/* ****************************************************************************************************************** */
// region: Helpers
/* ****************************************************************************************************************** */

/**
 * Patches existing Compiler Host (or creates new one) to allow feeding updated file content from cache
 */
function getPatchedHost(
  maybeHost: CompilerHost | undefined,
  tsInstance: typeof ts,
  compilerOptions: CompilerOptions
): CompilerHost & { fileCache: Map<string, SourceFile> }
{
  const fileCache = new Map();
  const compilerHost = maybeHost ?? tsInstance.createCompilerHost(compilerOptions, true);
  const originalGetSourceFile = compilerHost.getSourceFile;

  return Object.assign(compilerHost, {
    getSourceFile(fileName: string, languageVersion: ts.ScriptTarget) {
      fileName = tsInstance.normalizePath(fileName);
      if (fileCache.has(fileName)) return fileCache.get(fileName);

      const sourceFile = originalGetSourceFile.apply(void 0, Array.from(arguments) as any);
      fileCache.set(fileName, sourceFile);

      return sourceFile;
    },
    fileCache
  });
}

// endregion


/* ****************************************************************************************************************** */
// region: Program Transformer
/* ****************************************************************************************************************** */

export default function transformProgram(
  program: Program,
  host: CompilerHost | undefined,
  config: PluginConfig,
  { ts: tsInstance }: ProgramTransformerExtras,
): Program {
  const compilerOptions = program.getCompilerOptions();
  const compilerHost = getPatchedHost(host, tsInstance, compilerOptions);
  const rootFileNames = program.getRootFileNames().map(tsInstance.normalizePath);

  /* Transform AST */
  const transformedSource = tsInstance.transform(
    /* sourceFiles */ program.getSourceFiles().filter(sourceFile => rootFileNames.includes(sourceFile.fileName)),
    /* transformers */ [ transformAst.bind(tsInstance) ],
    compilerOptions
  ).transformed;

  /* Render modified files and create new SourceFiles for them to use in host's cache */
  const { printFile } = tsInstance.createPrinter();
  for (const sourceFile of transformedSource) {
    const { fileName, languageVersion } = sourceFile;
    const updatedSourceFile = tsInstance.createSourceFile(fileName, printFile(sourceFile), languageVersion);
    compilerHost.fileCache.set(fileName, updatedSourceFile);
  }

  /* Re-create Program instance */
  return tsInstance.createProgram(rootFileNames, compilerOptions, compilerHost);
}

// endregion


/* ****************************************************************************************************************** */
// region: AST Transformer
/* ****************************************************************************************************************** */

/**
 * Change all 'number' keywords to 'string'
 *
 * @example
 * // before
 * type A = number
 *
 * // after
 * type A = string
 */
function transformAst(this: typeof ts, context: TransformationContext) {
  const tsInstance = this;

  /* Transformer Function */
  return (sourceFile: SourceFile) => {
    return tsInstance.visitEachChild(sourceFile, visit, context);

    /* Visitor Function */
    function visit(node: Node): Node {
      if (node.kind === SyntaxKind.NumberKeyword)
        return context.factory.createKeywordTypeNode(tsInstance.SyntaxKind.StringKeyword);
      else
        return tsInstance.visitEachChild(node, visit, context);
    }
  }
}

// endregion

[nonara]

@craigphicks There are a few key differences in a transformed SourceFile and an original. It's been a little bit since I've been in that part of the compiler, so I'll answer to the best of my memory and knowledge.

A transformed SourceFile is produced during the emit phase. As a result, it's more of a facsimile or hybrid of a SourceFile, as far as the rest of the compiler is concerned. Before the transformation process, I recall that certain properties are stripped, in preparation. Additionally, it lacks position information for nodes which correlate to original source text (as it would, because they no longer correlate). The source text is also never updated. Each updated node contains a sort of chain, going back to the original node.

Most germane to the original intent, I believe that passing a transformed SourceFile directly to Program will fail to walk the types for the updated source, as there is (partial) existing type data for that file.

That aside, I believe there are various other ways it will break down, given the earlier issues I mentioned. I imagine it would also be bad from a garbage collection perspective, as there are many references and links associated with the former Program, which you'd like to throw away. These, also, will probably cause issues, as you want everything in a Program to be linked to the same. I've experienced a lot of problems related to that sort of thing while investigating routes for this. For my purposes, the right way to go was to produce a set of pristine SourceFile, which is what the compiler API is expecting.

You're welcome to test this and look further into it, however. Again, this is from my recollection, and it was a while back, so some of it may be inaccurate. I believe the simple answer is that these stripped / hybrid files aren't meant to be used outside of the emit cycle, and will therefore produce side effects.

If you look into it further and discover anything which confirms or refutes anything I've said, please do share in this thread!

[zhfnjust]

this work fine except debugging. i cannot debug my *.ts Program after transformed。

When starting the debugger, it always automatically stops at the last line of the generated js file no matter where I set the breakpoint.

Here is the minimal codebase to reproduce the problem：

https://github.com/zhfnjust/plugin-ts

[bogeeee]

Had modified you code, cause i forgot to set "transformProgram": true.
But here's a small impovement for that, to give a user friendly error then.

[bogeeee]

Got the example working (and thanks for this) and the visitor function replaces a single symbol. But i can't navigate through the AST from there: node.parent is undefined, node.childNodes is undefined (getChildNodes() throws an error), the tsInstance.is... methods throw an error, ...
Contrary to the visitor function in your source transformer example, where this is all working fine. Can you have a look at that ?
Here's the repo where your example is set up.

[bogeeee]

Can confirm that source maps are screwed up in my project as well. Even if the transformer does not insert a single statement.
@zhfnjust: Did you find a solution so far ?