Repository navigation
Conversation
Main sources still compile against empty trees and symbols. Tests are expected to fail until a semsitter parser fills them in.
Run one SemSitter scan (with the classpath) and its semantic dump over the batch, then parse each file with tree-sitter-java and attach symbols, resolved occurrences, graph call targets and graph facts by byte span.
…uery it Rust: a serve mode answering analyze, cst, parse and quit requests, and cst and parse commands printing the CST of files. Java: SemsitterBridge client.
… wire the parser
A new expression whose type reference is not a constructor used to leave a class or enum-field symbol on the identifier, and rules that cast it aborted the analysis.
…range Leaf primitive types and braces nested under a block were turned into tokens at the wrong place, so highlighting and issue ranges fell outside the line.
scoped_type_identifier was not a type kind, so a generic type like Map.Entry<K, V> reached type() with a null child and the translator threw.
Top-level methods were emitted as compilation-unit members, so visitors that look up the enclosing class crashed on Java 25 compact files.
The scanner drops issues whose location has no text range, which discarded every file-level finding such as S105.
Catch-clause variables had no "type" field in the CST, so every catch parameter fell back to an inferred "var" token at line 1, corrupting highlighting and file offsets downstream. Multi-catch types are now translated as a union type. Type parameters were also registered in the declarations map used by Symbol.declaration(). A field or variable typed with a type parameter resolves its type symbol to that type parameter, and TypeSymbol.declaration() unconditionally casts to ClassTree, so any check reading annotations on such a type crashed. The ECJ-based parser never declared type-parameter bindings; do the same here.
Member lookup matched a call's actual argument count against a method's declared parameter count exactly, so any varargs call passing more arguments than declared parameters (e.g. String.format with several arguments) failed to resolve, leaving parameterTypes() empty and crashing every check that indexes into it. Matching now falls back to a varargs-aware comparison when no exact-arity member is found. The semsitter bridge's type dump never reports whether a method is varargs, so isVarArgsMethod() stayed false even for genuine varargs library methods. It's now inferred from the declaration's last parameter (ellipsis token) when available, or from the last parameter type being an array otherwise.
A variable or field typed with a type parameter resolves its type symbol to that type parameter's own TypeSymbol. declaration() unconditionally cast the stored declaration tree to ClassTree, which crashed for such symbols. Now declaration() checks the tree type and returns null instead of casting blindly, matching the API's @nullable contract.
… empty statements
…evel semicolons
…heritance cycles
The pipe only carries a ping byte so the frontend blocks until the document is ready, and the CST, semantic dump and graph facts stay in the shared mapping. --dump-json still writes them as JSON for debugging.
The UCFG classpath moved to JVM settings upstream and was never read here because UCFG emission is off. SemSitter now resolves the call in the CST test on its own, so the reference no longer comes from the call-graph fallback.
…he enclosing class of an inner class
| if (request.has("javaLevel") && !settings.has("jvm.java_version")) { | ||
| JsonArray level = new JsonArray(); | ||
| level.add(request.get("javaLevel").getAsString()); | ||
| settings.add("jvm.java_version", level); | ||
| } |
There was a problem hiding this comment.
⚠️ Performance: Background module scan is discarded once javaLevel is in the settings
prepareAsync (AnalyzerSession.java:97-116) builds its request without javaLevel, and publishModule in JavaFrontend never adds jvm.java_version to the module settings. So the background scan stores typesSettings with no Java version. When sonar.java.source is set (most real projects), the first batch request goes through SemsitterBridge.analyze(..., javaLevel), and settings() now adds jvm.java_version (lines 258-262). The settings key no longer matches, so analyze calls stop(); start(root) and runs full-gen again. That throws away the scan prefetched ahead of the first file and does the module scan twice. Before this commit the prepare and batch settings were the same. Fix: send the same level from the prepare path, for example by publishing jvm.java_version from sonar.java.source in publishModule (the module setting wins over javaLevel), or by adding javaLevel to the prepareAsync request.
Publish the Java level with the module settings so the prepare scan and the batch requests share a settings key:
config.get("sonar.java.source").map(String::trim).filter(v -> !v.isEmpty())
.ifPresent(v -> settings.put("jvm.java_version", List.of(v.startsWith("1.") ? v.substring(2) : v)));
- Apply fix
Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎
| public static JavaTree.CompilationUnitTreeImpl parse(String version, String unitName, String source) { | ||
| return parse(version, unitName, source, List.of()); | ||
| } | ||
|
|
||
| /** {@link #parse(String, String, String)} with a classpath, as ECJ attributed single sources. */ | ||
| public static JavaTree.CompilationUnitTreeImpl parse(String version, String unitName, String source, List<File> classpath) { | ||
| Objects.requireNonNull(version); | ||
| Objects.requireNonNull(unitName); | ||
| Objects.requireNonNull(source); | ||
| Path directory = null; | ||
| try { | ||
| SemsitterBridge bridge = unitBridge(); | ||
| directory = privateTempDirectory(); | ||
| Path file = directory.resolve(Path.of(unitName.endsWith(".java") ? unitName : (unitName + ".java")).getFileName().toString()); | ||
| Files.writeString(file, source, StandardCharsets.UTF_8); |
There was a problem hiding this comment.
⚠️ Cross-Repo: SE test helper JParserTestUtils.parse won't compile against this frontend
Fix here and in SonarSource/sonar-java-symbolic-execution
Breaks in: SonarSource/sonar-java-symbolic-execution - java-symbolic-execution/java-symbolic-execution-plugin/src/test/java/org/sonar/java/se/utils/JParserTestUtils.java - JParserTestUtils.parse
Breaks in: SonarSource/sonar-java-symbolic-execution, in java-symbolic-execution/java-symbolic-execution-plugin/src/test/java/org/sonar/java/se/utils/JParserTestUtils.java (JParserTestUtils.parse).
That helper builds an ECJ parser with JParserConfig.Mode.FILE_BY_FILE.create(version, classpath).astParser() and passes it to the static JParser.parse(parser, version, unitName, source). This PR turns JParser into a final utility class. Its only single-source entry points are parse(String version, String unitName, String source) and parse(String, String, String, List<File>) (JParser.java:46-93), and both drive the SemSitter analyzer. The re-added JParserConfig has no astParser() method, and ECJ is no longer how parsing happens. So once the SE repo bumps its pinned java-frontend to a build with this change, its test helper no longer compiles and no SE test can build. The master base already lacked these APIs, but the SE repo still pins a release that has them. Dropping ECJ is the point of this PoC, so the consumer is the side that has to move. Even after it moves, its tests need an analyzer executable, either the packaged one, -Dsonar.java.semsitter.executable, or SEMSITTER_ANALYZER.
Here: Before releasing, list the removal of JParserConfig#astParser() and of the static JParser.parse(ASTParser/JParser, …) in the release or migration notes. Name JParser.parse(version, unitName, source, classpath) as the replacement, and note that MAXIMUM_SUPPORTED_JAVA_VERSION lives on JParserConfig. Coordinate with sonar-java-symbolic-execution so it does not bump java-frontend before it has migrated.
There: In JParserTestUtils.parse, remove the astParser() call and call the four-argument string-based JParser.parse with the classpath. Make sure the test JVM can find a SemSitter analyzer (the packaged binary or sonar.java.semsitter.executable).
In SonarSource/sonar-java-symbolic-execution - java-symbolic-execution/java-symbolic-execution-plugin/src/test/java/org/sonar/java/se/utils/JParserTestUtils.java:
public static CompilationUnitTree parse(String unitName, String source, List<File> classpath) {
JavaVersion version = JParserConfig.MAXIMUM_SUPPORTED_JAVA_VERSION;
return JParser.parse(version.effectiveJavaVersionAsString(), unitName, source, classpath);
}
Was this helpful? React with 👍 / 👎
| import org.sonar.java.semsitter.SemsitterBridge; | ||
|
|
||
| public interface JParser { | ||
| public final class JParser { |
There was a problem hiding this comment.
ℹ️ Cross-PR: JDK release option from #6459 loses its consumer and stops applying
Breaks when #6459 merges to master, this PR is rebased on top of it, and a project sets sonar.java.useJdkReleaseOption to analyze for an older Java release.
How
- #6459 sends
shouldUseJdkReleaseOptionthroughJParser.Mode.create(..., useRelease)andJDTParser.create(...)(java-frontend/src/main/java/org/sonar/java/model/JParser.java:34on #6459). - This PR turns
JParserinto a final utility class with noMode(java-frontend/src/main/java/org/sonar/java/model/JParser.java:37) and deletes theorg.sonar.java.jdtpackage.JParserConfig.Mode.createtakes only(javaVersion, classpath[, ignoreSplit])(JParserConfig.java:72-81). JParserConfig.bridgeClasspathputs the running JDK'slib/modulesfirst, andopenBridgesends only a numeric Java level (JParserConfig.java:225-254). Nothing restricts the JDK API to the target release, so classes newer than the configured release stay visible to the rules.
Fix
- This PR: Carry
shouldUseJdkReleaseOptionintoJParserConfig.Mode.createand pass it to the analyzer (for example ajvm.*release setting), or record that the option is unsupported under SemSitter. - #6459: Nothing is needed before this PoC goes further. Keep the option's contract documented so the SemSitter path can implement it.
Compared 824981b with #6459 at 822819f
Was this helpful? React with 👍 / 👎
| void openBridge(List<? extends InputFile> files) { | ||
| try { | ||
| bridge = SemsitterBridge.start(JParser.bridgePath()); | ||
| if (!files.isEmpty()) { | ||
| Path root = files.get(0).path().toAbsolutePath().getParent(); | ||
| bridge.analyze( | ||
| root == null ? Path.of(".") : root, | ||
| files.stream().map(file -> file.path().toAbsolutePath()).toList(), | ||
| bridgeClasspath(classpath), | ||
| javaVersion.isNotSet() ? Integer.MAX_VALUE : javaVersion.asInt()); | ||
| } | ||
| } catch (Exception failure) { | ||
| closeBridge(); | ||
| LOG.warn("SemSitter bridge did not start ({})", failure.toString()); | ||
| } |
There was a problem hiding this comment.
ℹ️ Cross-PR: Sourcepath files from #6501 never reach the server's scan
Breaks when both PRs merge with this PR's AnalyzerSession replacing #6501's bridge, and sonar.java.internal.sourcepath names sources that are not analyzed.
How
- #6501 adds the sourcepath
.javafiles to thefilespassed tobridge.analyzeinopenBridge(java-frontend/src/main/java/org/sonar/java/model/JParserConfig.java:224on #6501). - In this PR,
JavaFrontendpublishesSemsitterBridge.scanFiles(allFiles)and a module root (JavaFrontend.java:168,269). AnalyzerSession.analyzescansscanFilesand falls back torequest.filesonly whenscanFilesis empty and no module root is set (AnalyzerSession.java:145-152). The extra sourcepath files are dropped, so project types they declare stay unresolved.
Fix
- This PR: Merge the sourcepath files into the list given to
SemsitterBridge.scanFiles, or haveanalyzeunionrequest.fileswithscanFiles. - #6501: Port to the JSON
AnalyzerSessionAPI: the 4-argumentanalyzeandbridgeClasspath, with sourcepath files sent throughscanFiles.
Compared 824981b with #6501 at 3e91a4d
Was this helpful? React with 👍 / 👎
| // a parameterized return keeps its arguments, so the member's return type is the one a call reports | ||
| if (!method.isStatic() && retType.has("args")) { | ||
| JType plain = typeFromFact(retType, Map.of(), true); | ||
| if (!plain.isUnknown() && plain.isParameterized() | ||
| && plain.args.stream().noneMatch(argument -> argument.isCapture() || argument.isWildcard())) { | ||
| method.returnTypeSymbol(plain.symbol()); | ||
| } | ||
| } |
There was a problem hiding this comment.
⚠️ Bug: Library return type now keeps unbound type variables such as List<E>
The new block in memberFromTypeInfo copies the invocation-time rule at JSema.java:742-748 but leaves out its !mentionsTypeVariable(plain) guard. It only rejects wildcard and capture arguments. So any instance library method whose signature names a class or method type variable now has a parameterized return type that carries a free variable: List.subList returns List<E>, Map.entrySet returns Set<Map.Entry<K,V>>, and Stream.map returns Stream<R>. Before this change the return type was the erased descriptor type.
That stored type reaches call sites through the fallback at line 750 of invocationReturnType. If the receiver is raw but not written raw (an unqualified subList(0, 1) in class Foo extends ArrayList, or a qualifier returned by another call), classSubstitution is empty and SemsitterTranslator does not call rawReturn, so the call now evaluates to List<E> instead of List. The same fallback also returns the leaking type (e.g. Stream<R>) whenever method type inference fails, because the earlier branch rejected it as leaks. A free type variable in an expression type is new, incorrect behavior that checks will see.
Fix: add the same !mentionsTypeVariable(plain) guard used at line 745, so that only fully concrete arguments, like the List<String> in the new test, are kept.
Keep the declared arguments only when they are concrete, matching the invocation-time rule.:
if (!method.isStatic() && retType.has("args")) {
JType plain = typeFromFact(retType, Map.of(), true);
if (!plain.isUnknown() && plain.isParameterized() && !mentionsTypeVariable(plain)
&& plain.args.stream().noneMatch(argument -> argument.isCapture() || argument.isWildcard())) {
method.returnTypeSymbol(plain.symbol());
}
}
- Apply fix
Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎
…as no single abstract method
…y bound in conditionals
| private boolean takesNoFunction(Symbol.MethodSymbol method, Set<Integer> functions) { | ||
| List<Type> parameters = method.parameterTypes(); | ||
| for (int index : functions) { | ||
| if (index >= parameters.size()) { | ||
| continue; | ||
| } | ||
| Type target = parameters.get(index).erasure(); | ||
| if (target.isPrimitive() || target.isArray() || (target.isClass() && !target.isUnknown() && !target.isTypeVar() && !target.symbol().isUnknown() | ||
| && (!target.symbol().isInterface() || (declaresType(target.fullyQualifiedName()) && functionalMethod(target) == null)))) { | ||
| return true; | ||
| } | ||
| } | ||
| return false; | ||
| } |
There was a problem hiding this comment.
⚠️ Bug: Lambda overload filter drops functional interfaces with default overrides
takesNoFunction now throws out any declared interface for which functionalMethod returns null. collectAbstractMethods only ever adds abstract methods with putIfAbsent. It never lets a default method in a subinterface hide the abstract method it overrides from a parent. Take Guava's Predicate<T>: it declares abstract apply(T) and overrides java.util.function.Predicate.test(T) with a default method. The code ends up collecting both apply/1 and test/1, so functionalMethod returns null and the interface is treated as non-functional.
As a result, a call like Iterables.filter(list, x -> ...) loses its Predicate overload (the Class<T> overload is dropped as well). declaredMethod then returns null and a call that used to resolve is now unresolved. Before this commit an interface parameter was never filtered out.
Fix: when collectAbstractMethods walks the hierarchy, skip any name/arity key that a nearer (sub)interface already declares as a non-abstract (default) method. Alternatively, only filter the candidate when the interface has no abstract methods at all.
Record keys of default methods met first so inherited abstract methods they override are not counted:
private void collectAbstractMethods(String ownerFqn, Set<String> seen, Map<String, JMethodSymbol> found, Set<String> overridden) {
...
if (memberFromTypeInfo(typeSymbol(ownerFqn), element.getAsJsonObject()) instanceof JMethodSymbol method && !method.isUnknown()) {
String key = method.name() + "/" + method.parameterTypes().size();
if (!method.isAbstract()) { overridden.add(key); }
else if (!overridden.contains(key) && !OBJECT_METHODS.contains(key)) { found.putIfAbsent(key, method); }
}
...
}
- Apply fix
Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎
| @Test | ||
| void constantOfAFieldFromTypeInfoIsKept() { | ||
| sema.ingestDocument(JsonParser.parseString(""" | ||
| {"symbols": [ | ||
| {"id": "p.Cfg", "kind": "type", "name": "Cfg"}, | ||
| {"id": "p.Cfg#NAME", "kind": "field", "name": "NAME", "owner": "p.Cfg"}]} | ||
| """).getAsJsonObject()); | ||
| JTypeSymbol owner = (JTypeSymbol) sema.resolve("p.Cfg"); | ||
| JsonObject member = JsonParser.parseString(""" | ||
| {"id": "p.Cfg#NAME", "name": "NAME", "k": "f", "constant": "abc"} | ||
| """).getAsJsonObject(); | ||
|
|
||
| Symbol field = sema.memberFromTypeInfo(owner, member); | ||
|
|
||
| assertThat(((Symbol.VariableSymbol) field).constantValue()).isPresent(); |
There was a problem hiding this comment.
💡 Quality: Test's second assertion never exercises servedConstant
In constantOfAFieldFromTypeInfoIsKept, the first call sema.memberFromTypeInfo(owner, member) already sets the constant on the cached p.Cfg#NAME symbol through field.constantValue(...). resolve then returns that same instance from symbols. The later ingestTypes + resolve("p.Cfg#NAME").constantValue() assertion passes on the cached value without ever reaching JVariableSymbol.constantValue() → servedConstant. The comment says this part tests "a field only ever read through its symbol", so that path is untested. Fix: move the second part into its own test with a fresh JSema and don't call memberFromTypeInfo first.
Was this helpful? React with 👍 / 👎
…ed as an argument
… the function's first parameter
Code Review
|
| Auto-apply | Compact | Unblock |
|
|
|
Was this helpful? React with 👍 / 👎 | Gitar
Duplicate of #6514. Same PoC, rebased onto the current
rombirli/semsitter-ecj-replacement-v2.antonio/poc-semsitter-udg-serveris unchanged.Draft PoC. Do not merge.
The 52 commits that start after
10fa6fd1f7are replayed on top of the updated base. Description and measurements stay on #6514.Summary by Gitar
AnalyzerSessionclient that keeps one SemSitter server process per module and caches type lookupsFlatFramedecoder for the flatget-fileframe, used from protocol 1.8JSemaandSemsitterTranslatorwith project member annotations, thrown types, and undeclared-type markingCompilerWarningswith redundant-cast and unreachable-catch detectionJDTParser,JDTAstConverter) with SemSitter-backedJParserConfigbatch parsingThis will update automatically on new commits.