Repository navigation
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #421 +/- ##
==========================================
- Coverage 92.88% 92.80% -0.08%
==========================================
Files 31 31
Lines 3289 3072 -217
==========================================
- Hits 3055 2851 -204
+ Misses 234 221 -13 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
This is how benchmark results would change (along with a 95% confidence interval in relative change) if 0abd75b is merged into master:
Further explanation regarding interpretation and methodology can be found in the documentation. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
E_loo()gives anNAPareto k wherepsis()givesInffor the same log ratios. This occurs when many of the largest ratiosexp(log_ratios - max(log_ratios))underflow to 0, so that k cannot be estimated from the tail.psis()sets k toInfin this case.E_loo()estimates k from the tail of the same ratios a second time withposterior::pareto_khat(), which givesNAwith posterior 1.8.0.Example
Changes
With this PR,
E_loo()takes the k of the ratios frompsis_object, wherepsis()stored it, and estimates k only for the tails ofh * r. The k of the ratios then follows the rules ofpsis().psis()givesInfwhen it cannot estimate k from the tail. This occurs when the tail has fewer than 5 draws, when all tail values are equal, or when a quarter or more of the tail values are tied at the low end of the tail (as with the zeros from underflow). The PR also removes the code for posterior versions that returned a list (loo requires posterior >= 1.7.0) and the code for twoNAvalues.Other changes of the returned k:
log_ratiosofpsis_objectand a finitepsis()k, the k changes only by rounding (at most 3.6e-14 forexample_loglik_matrix()and for one example with a heavy tail).log_ratios, the k of the ratios is now thepsis()k of the raw ratios. Before, it was the k of the smoothed weights. For the 32 columns ofexample_loglik_matrix()andtype = "quantile", the change of the k is between -0.043 and 0.068. The k ofh * rstill uses the smoothed weights in this case.E_loo()now givesInf, aspsis()does. Before, it gave the k ofhfor a continuousx, andNAfor a binaryx.E_loo()now givesInf, aspsis()does. Before, it gaveNA(tested with S = 20).loo_approximate_posterior(), the k of the ratios is now the k of the corrected ratios that the weights use. Before, it was the k of thelog_ratiosargument.loo_moment_match()replaces the weights and the k in a saved psis object for the observations that it improves.E_loo()withxfrom the original draws is not valid for these observations, before and after this PR. With the originallog_ratios,E_loo()gave the high k of the original ratios for them. It now gives the moment-matched k.The k of
h * rThis PR changes also the scaling of
h * rbefore callingposterior::pareto_khat.E_loo()now scalesh * ron the log scale so that the largest|h * r|is 1. The k ofh * rthen does not change whenxis multiplied by a constant. On master,E_loo()scales only the ratios so that the largest ratio is 1. For1e-307 * xand the 32 columns ofexample_loglik_matrix(),E_loo()on master returns k =Inffor each column, becauseh * rhas subnormal values.This PR does not change the rule for the k that
E_loo()returns.E_loo()estimates k for the left tail and for the right tail ofh * rwithposterior::pareto_khat(). For a tail, k cannot be estimated when all tail values are equal, or when a quarter or more of them are tied.E_loo()returns the largest of the k of the ratios and the k values of the two tails ofh * r, and leaves out a tail for which k cannot be estimated. If k cannot be estimated for either tail ofh * r,E_loo()returns the k of the ratios.The table shows examples with S = 1000 and a tail of 95 draws (posterior 1.8.0). In rows 4 to 6, only 150, 100 and 30 ratios are nonzero; the other ratios underflow to 0. Only row 6 changes with this PR.
psis())h * r, left tailh * r, right tailE_loo()k, masterE_loo()k, this PRxx, 62% zerosxis 0 for 95% of the drawsIn the
NAcells of the table, a quarter or more of the tail values are exact zeros. In rows 2 and 3, exact zeros ofxcause the ties, andE_loo()uses the other tail (row 2) or the k of the ratios (row 3). In rows 4 and 5, the underflow of the ratios causes the ties.h * rthen has fewer nonzero values on each side of 0 than the tail has draws. The k of the ratios is still finite in these rows. With fewer zeros in the tail, k can be estimated and is high (row 4, left tail). TheE_loo()k is thus not monotone in the number of nonzero ratios. This PR does not change the rule for when k can be estimated.The documentation of
pareto_know says where the k of the ratios comes from. The documentation oflog_ratiosnow says that they are the log ratios thatpsis()used to computepsis_object.Tests
New tests in
tests/testthat/test_E_loo.Rcover:x,type = "quantile", nolog_ratios);h * r;psis()k for quantiles and a binaryx;xscaled by 1e-307 (added to the existing test of the scale ofx).On master, the new tests give 16 failures. The full test suite passes (1365 expectations) with R 4.6.2 and posterior 1.8.0.
This PR was assisted by Claude.