Skip to content

Complete quoted matches in one Tab and close the quote on a menu choice - #1831

Open
tleonhardt wants to merge 9 commits into
mainfrom
quote_completion_combined
Open

tleonhardt wants to merge 9 commits into
mainfrom
quote_completion_combined

Conversation

@tleonhardt

@tleonhardt tleonhardt commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Closes #1821.

Fixes the two problems that issue tracks. Both affect any completion which needs quotes, such as a file name with a space, with any command which completes paths or choices.

Changes

One Tab is enough when a match needs an opening quote

Previously the first Tab only inserted the quote and a second Tab was needed to complete the name:

(Cmd) edit my<TAB>     -> edit "my
(Cmd) edit "my<TAB>    -> edit "my file.txt"

Cmd2Completer.get_completions() inserted the quote into the buffer and returned no matches, expecting prompt-toolkit to start a new completion. It never did. prompt-toolkit discards a completion whose input changed while it ran, and it only starts another one when text was added to the end of the input. The quote goes in front of the search text.

Now:

  • get_completions() leaves the buffer alone during the completion which finds that a quote is needed. When prompt-toolkit has finished that completion, the quote is inserted on the event loop and completion is started again with the quote in place, so the common prefix of the matches can still be inserted.
  • The matches are passed to that second completion in its CompleteEvent, so Cmd.complete() runs once for one Tab and a completion table is printed once.
  • Nothing is inserted if the completion was cancelled or the input changed in the meantime.
  • The buffer is no longer edited from the completion thread.
  • If the buffer completes while typing, inserting the quote doesn't start a completion of its own, which prompt-toolkit would run in place of the one cmd2 starts. The buffer's other listeners are told about the quote as usual, so auto-suggest is asked about the new text.
(Cmd) edit my<TAB>     -> edit "my file.txt"
(Cmd) edit lo<TAB>     -> edit "long name      (menu opens with both matches)

Only the completion key's request is started again

prompt-toolkit doesn't tell a completer what a request will do with its matches. Starting again always asks for the common prefix to be inserted, which is what Tab does, so it is only right for a request which comes from Tab.

  • cmd2 now binds Tab itself, in place of prompt-toolkit's binding for it, whatever completekey is set to. It does what that binding does, and marks its request with a CompleteEvent of its own so the completer can recognize it. A custom completekey is bound alongside Tab, as before.
  • The binding steps aside when the session shows completions like readline (CompleteStyle.READLINE_LIKE), which has a Tab binding of its own.
  • Every other request which prompt-toolkit runs gets the opening quote as part of each match and goes on to do what it was started for. That covers Ctrl-N and Ctrl-P in Vi insert mode, Meta-/ in Emacs mode, and a custom completekey.
  • The key binding which lists completions like readline asks for them directly. In a session with that style, it gets the quote put in the buffer right away and the matches for the quoted text. It reads the buffer again once it has the completions, so the text the matches share is inserted with the quote in place. Other bindings which ask directly, such as Meta-* in Emacs mode which inserts every match, get each match with its own quotes.

A match chosen from the menu closes its quote

Previously the closing quote was only added for a single match, so choosing one from the menu and running the command failed:

(Cmd) edit "sp ace.txt
Invalid syntax: No closing quotation

Now each match offered in a menu at the end of the line closes an open quote and adds a trailing space, like a single match does. A match which the user will keep completing does neither, so the rest of it can still be completed:

(Cmd) edit "sp ace.txt"
(Cmd) edit "dd one/
  • path_complete() marks directories with a new private field, CompletionItem._allow_finalization. A trailing separator can't be used to recognize them, because a directory is offered without one when a separator already follows the cursor. The field is not part of an item's equality or hash.
  • delimiter_complete() marks each match which has more segments in the same way. A finished match offered alongside one, such as some file next to some dir/, closes its quote.
  • A match marked this way is not finalized when it is the only match either. That matters to a completer which filters the results of those two functions down to one directory-like match.

In the middle of a line, a closing quote is only added where it is certain to be needed

A completion only replaces text before the cursor, so the argument may carry on after it. Whether a quote character further along closes this argument, or belongs to another argument, or is an apostrophe or an inch mark, can't be told. An earlier version of this PR tried to work that out, and each revision got some line wrong which main gets right (see "Fixes made during review"). So nothing depends on it now:

  • The only match always closes its quote, as on main, unless the next character is already that quote.
  • A match chosen from the menu closes its quote at the end of the line, or when cmd2 opened the quote around a word which ended at the cursor. Anywhere else nothing is added, as on main.
only match
(Cmd) edit "my|"                     -> edit "my file.txt"                    (main: edit "my file.txt"")
(Cmd) edit my| 5" nail               -> edit "my file.txt" 5" nail
(Cmd) shell cat "my| --title="x"     -> shell cat "my file.txt" --title="x"

chosen from the menu
(Cmd) edit sp| other                 -> edit "sp ace.txt" other               (main: edit "sp ace.txt other)
(Cmd) edit "sp|.bak"                 -> edit "sp ace.txt.bak"
(Cmd) shell cat "sp| backup".txt     -> shell cat "sp ace.txt backup".txt
(Cmd) edit "sp| other                -> edit "sp ace.txt other                (unclosed, as on main)

In 17 lines taken from the reviews, main gets 12 right and this gets 15, and it is never worse than main. The two it gives up are the last line above and edit "sp| "spare.txt": a quote which the user typed and which is not closed anywhere, completed from the menu in the middle of a line.

Documentation and tests

  • New Quotes Added by Completion section in the completion docs, and two CHANGELOG entries under 4.4.4. The CHANGELOG also says that Completions.allow_finalization now applies when there are several matches.
  • New tests/test_completion_keys.py presses keys at a prompt which reads from a pipe, so the key bindings, cmd2's completion logic, and prompt-toolkit's handling of the results are tested together. It covers one Tab for a single match (with and without complete_in_thread), the shared prefix, shell arguments, read_input(), a custom completekey, Ctrl-P in Vi insert mode, shortcuts, directories, delimiter_complete(), completing in the middle of a line in front of a closing quote, another quoted argument, an apostrophe, an inch mark or a redirection, a readline-style session, the history search, and a prompt which ends while a completion is running. These tests are skipped on Windows, like the other tests which use a pipe for input.
  • New unit tests in tests/test_pt_utils.py run prompt-toolkit's real completion code. They cover a request which prompt-toolkit is still waiting on, one which is cancelled or outdated while cmd2 is completing, one which is cancelled just before prompt-toolkit finishes it, requests which don't come from the completion key, a buffer which completes while typing, auto-suggest, the reuse of the matches, and other listeners on the buffer's event.

Fixes made during review

These are in the second to ninth commits.

  • Closing quotes in the middle of a line. The first version closed the quote of every match chosen from the menu, which doubled a closing quote that was already there: edit "sp|" became edit "sp ace.txt"". Later revisions tried to work out from the rest of the line whether the quote was already closed: counting quote characters, skipping the ones inside other quoted arguments, reading the line with the parser's rules, treating redirections as the end of a word, and telling apostrophes from quotes. Each one fixed the lines reported against it and got another wrong, among them edit "sp.bak", shell cat my 'a"b', edit "sp 'copy", echo 'my don't, edit my 5" nail, shell cat "sp.bak">out, shell cat "sp backup".txt and shell cat "my --title="x". The ninth commit replaces all of that with the two rules above, which get every one of those lines right.
  • delimiter_complete() kept every match open. It returned allow_finalization=False for the whole result when any match had more segments, so pick "some file stayed open because some dir/ was also offered.
  • Matches from an abandoned request could be reused. They were cached on the completer before checking whether the input had changed. They now travel with the restarted completion.
  • A cancelled completion still inserted the quote. A cleared completion state was taken to mean that prompt-toolkit had finished the request, but cancelling clears it too. The request now only counts as finished if prompt-toolkit was seen to reach its end with the request's state intact, which a cancelled or outdated request never is. A completion which runs in a thread waits (for up to a second) for the event loop to be watching the request before it returns, so prompt-toolkit can't finish it first.
  • The restart changed what other requests do. Ctrl-P in Vi insert mode asks for the last match to be selected, but the restart left nothing selected. Bindings which ask for completions directly got none at all. See "Only the completion key's request is started again".
  • Tab after an accepted menu choice completed against the closing quote. edit "sp ace.txt" followed by Tab gave edit "sp ace.txt"-h. The trailing space prevents that.
  • Completing while typing took over the restart. With complete_while_typing=True, inserting the quote started an automatic completion which prompt-toolkit ran in place of the one cmd2 started, so Tab left edit "my.
  • Readline-style sessions lost their Tab binding. cmd2's binding took precedence over prompt-toolkit's binding for that style.
  • Readline-style completion made no progress on a prefix which needs a quote. The matches came back with the quote in front, which changes text before the cursor, so the binding only listed them and left edit lo as it was. It now gets edit "long name in one Tab.
  • A single unfinished match was finalized. delimiter_complete() no longer returns allow_finalization=False for several matches, so a completer which filtered its results down to one unfinished match got a trailing space after it.
  • Meta-* got the quote in the buffer and matches without one. That Emacs binding asks for completions directly, as the readline-style binding does, but inserts every match after the cursor. The result was shell cat "spsp ace.txt" spare.txt" . Putting the quote in the buffer is now limited to readline-style sessions.
  • Inserting only the quote could leave auto-suggest behind. With complete_while_typing on, the insertion didn't notify the buffer's listeners, so no new suggestion was asked for until the next key press.
  • Readline-style completion lost the quote after a shortcut. The quote goes after a shortcut, but the completions still replaced the shortcut and the quote, so @my became @my file.txt" .
  • Tab with a custom completekey only opened the menu. Tab was still prompt-toolkit's binding there, so for several matches which need a quote it inserted neither the quote nor the text they share.
  • The quote could be inserted into a prompt which had ended. If the line was submitted in the instant between prompt-toolkit finishing the request and the quote being inserted, the last redraw showed a quote which the returned line didn't have.
  • Event handler hygiene. The handler which waits for the request no longer removes itself while prompt-toolkit is calling the event's handlers, which made it skip the next one. It is also removed when the prompt ends while a completion is still running.

Behavior changes worth a look

  • cmd2 binds Tab. It replaces prompt-toolkit's binding for Tab and behaves the same way, including only being active while text can be inserted. This is the case whatever completekey is set to. An application which adds its own binding for Tab to the session's key bindings is affected by the order in which the two are added.
  • A match chosen from the menu at the end of the line now ends with a closing quote and a trailing space whenever a quote is open, including a quote the user typed. In the middle of a line it only closes a quote which cmd2 added around a word that ended at the cursor. A match which needs no quote is unchanged.
  • A single match no longer gets a closing quote when the next character is already that quote. It always got one before. In every other position it still does.
  • Completions.allow_finalization=False now also keeps the matches in a menu from closing a quote. It was ignored for multiple matches before.
  • delimiter_complete() returns allow_finalization=True for several matches even when some have more segments. It used to return False, which had no effect for several matches before this PR.
  • A match which path_complete() or delimiter_complete() marked as unfinished is not finalized even when it is the only match. Before, only Completions.allow_finalization decided that.
  • A custom completekey is unchanged: its first press opens the menu and inserts nothing, whether or not the match needs a quote. Tab behaves the same with or without one.
  • The restart depends on prompt-toolkit's behavior. It relies on prompt-toolkit notifying on_completions_changed when it has all the results of a request, then clearing the completion state and notifying again when there were none. If a future prompt-toolkit release changed that, Tab on a name which needs a quote would stop completing. The new unit tests run that code path, so CI would catch it.

Known limits

  • In the middle of a line, a quote which the user typed is left as it is when a match is chosen from the menu, so edit "sp| other stays unclosed. A single match inside a quoted word whose closing quote comes later still gets a second one, so edit "my|.bak" becomes edit "my file.txt".bak". Both are what main does.
  • A session set to CompleteStyle.READLINE_LIKE only lists matches that way on the first Tab. Cmd.complete() sets the session's style to one of the menu styles on every completion, as it does on main. That is not changed here.
  • A completer written outside cmd2 can't mark individual matches as unfinished, because the field is private. The docs and CHANGELOG now say that such a completer should return allow_finalization=False when any of its matches is unfinished, which keeps every match in the menu open, as before this PR. Making the field public would be an API addition.
  • The limits listed in Tab completion: opening quote needs a second Tab, and a menu choice leaves it unclosed #1821 (one quote character and one pair of quotes for all the matches being offered, and macros which run a shell command) are unchanged and are documented in Special Characters in Paths.

Testing

  • make check, make test (3188 passed, 3 skipped on macOS), and make docs-test pass locally.
  • Checked in a real terminal (tmux, macOS) that one Tab completes edit my, that a menu choice closes its quote and the command then runs, that a directory stays open and completion continues inside it, that the only match closes its quote in front of --title="x" and 5" nail, that a menu choice in the middle of a line closes the quote in edit sp| other and adds nothing in shell cat "sp| backup".txt and edit "sp| other, and that a custom completekey opens the menu on its first press while Tab still inserts the quote and the shared text.
  • Not checked in a real terminal on Windows. Vi key bindings, a readline-style session and complete_while_typing were only exercised by the tests.

Fixes the two problems tracked in #1821, which affect any match that
needs quotes, such as a file name with a space.

A second Tab was needed after the opening quote was inserted.
Cmd2Completer.get_completions() inserted the quote into the buffer and
returned no matches, expecting prompt-toolkit to start a new completion.
It never did: prompt-toolkit discards a completion whose input changed
while it ran and only restarts when text was added to the end of the
input, and the quote goes in front of the search text.

get_completions() now leaves the buffer alone during the completion
which found that a quote is needed. Once prompt-toolkit has finished
that completion, the quote is inserted on the event loop and completion
is started again with the quote in place, so the common prefix of the
matches can be inserted. The matches are kept for that second
completion, so cmd2's completion logic runs once and a completion table
is printed once. Nothing is inserted if the input changed in the
meantime. The buffer used to be edited from the completion thread.

Choosing a match from the menu left the quote unclosed, so the command
failed with "Invalid syntax: No closing quotation". Each match offered
in a menu now closes an open quote. A directory doesn't, so the rest of
the path can still be completed. path_complete() marks directories with
a private CompletionItem._allow_finalization field, because a directory
is offered without a trailing separator when one follows the cursor.

tests/test_completion_keys.py presses the completion key at a prompt
which reads from a pipe, so the key bindings, cmd2's completion, and
prompt-toolkit's handling of the results are tested together.

Co-authored-by: Kevin Van Brunt <kmvanbrunt@gmail.com>
@codecov

codecov Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.76%. Comparing base (4f55b15) to head (5b5c459).

Additional details and impacted files
@@           Coverage Diff            @@
##             main    #1831    +/-   ##
========================================
  Coverage   99.75%   99.76%            
========================================
  Files          23       23            
  Lines        6644     6750   +106     
========================================
+ Hits         6628     6734   +106     
  Misses         16       16            
Flag Coverage Δ
unittests 99.76% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Don't add a closing quote in front of one which is already there. A
completion only replaces text before the cursor, so with the cursor in
front of the closing quote of `edit "sp"`, choosing a match from the
menu gave `edit "sp ace.txt""`. A single match had the same problem
before this branch, and it is fixed the same way.

Let a finished match close its quote when delimiter_complete() offers
it alongside a match which has more segments. delimiter_complete() set
allow_finalization=False for the whole result if any match was
unfinished, which kept every match in the menu from closing the quote.
It now marks each unfinished match instead, as path_complete() does for
directories, and only returns allow_finalization=False for a single
unfinished match.

Cache the matches for the restarted completion only once it is certain
to start. They were cached before checking whether the input had
changed, so when the user edited the line and the restart was
abandoned, a later request for the same quoted text could reuse them.
Don't add a closing quote when text after the cursor already closes the
quote. The previous fix only looked at the character right after the
cursor, so with the cursor after `sp` in `edit "sp.bak"`, choosing a
match from the menu gave `edit "sp ace.txt".bak"` and split the
argument. A closing quote is now only added when the line needs one: an
odd number of the quote character after the cursor means the first of
them closes this quote and the rest are in pairs.

Don't insert the quote for a request which was cancelled. A cleared
complete_state was taken to mean that prompt-toolkit had finished the
request, but cancelling a slow completion clears it while the request
is still running. The quote was then inserted, prompt-toolkit ignored
the completion started after it, and the cached matches were left for
the next request. get_completions() now checks before it returns
whether its request is still the active one, and does nothing if it was
cancelled or the line was edited.

The matches are no longer cached on the completer. They are passed to
the restarted completion in its CompleteEvent, so they are dropped with
that request if prompt-toolkit doesn't run it and no other request can
be given them.
Start completion again after inserting an opening quote only for the
request which comes from the completion key. prompt-toolkit doesn't
tell a completer what a request will do with its matches, so starting
again as a request for the common prefix changed what every other
request does: Ctrl-P in Vi insert mode no longer selected the last
match, and bindings which ask for completions directly got none. cmd2
now binds the default completion key itself, in place of
prompt-toolkit's binding for Tab, and marks its request with a
CompleteEvent of its own. Every other request gets the opening quote as
part of each match and goes on to do what it was started for.

Don't start again for a request which was cancelled just before
prompt-toolkit finished it. The request is now only treated as finished
if prompt-toolkit was seen to reach its end with the request's state
intact, which a cancelled or outdated request never is. A completion
which runs in a thread waits for the event loop to be watching the
request before it returns, so prompt-toolkit can't finish it first.

Always close a quote which cmd2 opened. Nothing after the cursor can
close it, but a quote character in a later argument was taken for its
closing quote, so completing `my` in `shell cat my 'a"b'` left the
quote open. For a quote which the user typed, quote characters inside
an argument quoted with the other character are no longer counted.

Add a trailing space after a match which closes its quote in the menu.
Without it, pressing Tab after accepting the match completed a new
argument right against the closing quote.

Don't remove the event handler while prompt-toolkit is calling the
event's handlers, which made it skip the next one, and remove it when
the prompt ends while a completion is still running.

Document that a completer whose matches can be unfinished, like
directories, should return allow_finalization=False.
Replace the count of quote characters after the cursor, which misread
apostrophes and quotes in other arguments in both directions. A quote
is now taken to be closed already in these cases:

- The word under the cursor ends with the quote character, as in
  `edit "sp|.bak"`, or `edit sp|"` where the opening quote is missing.
- The user typed the opening quote, so the argument can have whitespace
  in it, and the rest of the line has a quote character at the end of
  an unquoted word, as in `edit "sp| 'copy"`. The rest of the line is
  read with the parser's own quoting rules, so the quotes of another
  quoted argument and an apostrophe in the middle of a word don't
  count.

A quote which cmd2 adds goes around what was one unquoted word, so
nothing past that word closes it. `edit my| 5" nail` and
`shell cat my| 'a"b'` both get their closing quote.

Don't let a buffer which completes while typing start a completion of
its own when the opening quote is inserted. prompt-toolkit ran that one
and ignored the completion which cmd2 started for the matches it had
found.

Leave Tab to prompt-toolkit when the session shows completions like
readline, which has a binding of its own for it.

Don't finalize the only match when it is marked as one the user will
keep completing. A completer which filters the results of
path_complete() or delimiter_complete() down to one directory-like
match got a trailing space after it.

Say in the CHANGELOG that Completions.allow_finalization now applies
when there are several matches.
Treat the redirection characters and the terminators as the end of a
word when deciding whether a quote is already closed, as the parser
does. With the cursor after `sp` in `shell cat "sp.bak">out`, the
closing quote was not seen because `.bak">out` has no whitespace in it,
and choosing a match added a second quote at the cursor. The text after
the cursor is now split by a function which follows the parser's
quoting rules and knows about punctuation, in place of shlex. A quote
right after a redirection character starts a quoted argument, so the
quotes of `>"out file"` are not taken for a closing quote.

Put the opening quote in the buffer right away when a key binding asks
for completions itself, as the binding which lists completions like
readline does. That binding reads the buffer again once it has the
completions, so it can insert the text the matches share with the quote
in place. It used to get each match with the quote in front, which
changes text before the cursor, so it only listed the matches and left
the line as it was. This is only done from the event loop, when
prompt-toolkit is not running a request and the buffer holds the line
being completed.
… case

Keep the opening quote when a key binding asks for completions itself
and the word follows a shortcut. The quote goes after the shortcut, but
the completions still replaced the shortcut and the quote, so in a
readline-style session `@my` became `@my file.txt" `.

Don't insert the quote into a prompt which has already ended. If the
line was submitted in the instant between prompt-toolkit finishing the
request and cmd2 inserting the quote, the last redraw showed a quote
which the returned line didn't have.

Bind Tab to cmd2's own request whatever the completion key is set to,
with a custom key bound alongside it. With a custom completekey, Tab
was still prompt-toolkit's binding, so it only opened the menu for
several matches which need a quote instead of inserting the quote and
the text they share. The custom key itself still only opens the menu on
its first press.
Take a quote character anywhere in an unquoted word for a closing
quote, not only one at the end of the word. With the cursor after `sp`
in `shell cat "sp backup".txt`, the quote which closes the quoted part
of the file name was missed and a second one was added at the cursor.
A single quote with a letter on each side is an apostrophe and is still
left alone, as in `don't`.

Only put the opening quote in the buffer for a request made directly by
a key binding when the session lists completions like readline. The
Meta-* binding of Emacs mode asks the same way but inserts every match
after the cursor, so it got the quote in the buffer and then matches
without one: `shell cat "spsp ace.txt"  spare.txt" `. It now gets each
match with its own quotes. The session's style is read before cmd2's
completion logic runs, since that sets the session to a menu style.

Tell the buffer's listeners when the opening quote is inserted, whether
or not the buffer completes while typing, so auto-suggest is asked
about the new text. Only the buffer's own completion while typing is
held off for that insertion, which is what took the place of the
completion cmd2 starts.
Replace the check which read the rest of the line to decide whether a
quote was already closed. Whether a quote character further along is
this argument's closing quote, part of another argument, an apostrophe
or an inch mark can't be told, and each revision of the check got some
line wrong which main gets right. The latest was
`shell cat "my| --title="x"`, which lost its closing quote.

A closing quote is now only added where it is certain to be needed, and
everywhere else the line is treated as it was before this branch:

- The only match always closes its quote, as on main, unless the next
  character is already that quote. That exception fixes `edit "my|"`
  becoming `edit "my file.txt""`.
- A match chosen from several closes its quote at the end of the line,
  or when cmd2 opened the quote around a word which ended at the
  cursor. Otherwise nothing is added, as on main.

This removes the argument splitter, the apostrophe rule and their table
of cases. In 17 lines taken from the reviews, main gets 12 right, the
previous check 16 and this one 15, and this one is never worse than
main. The two it gives up are a quote the user typed which is not
closed anywhere, completed from the menu in the middle of a line, such
as `edit "sp| other`. They are left unclosed as they are on main.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tab completion: opening quote needs a second Tab, and a menu choice leaves it unclosed

1 participant