Proposed clarification of spec for int/float/complex promotion

[JelleZijlstra]

Fixes #1746.

[erictraut]

Looks good to me. One minor suggested improvement in the test case, but it's fine without this change.

[hauntsaninja]

I think worth having an unguarded f.hex() as well

[carljm]

This one is interesting, because it's not totally clear to me what behavior we should specify for an unguarded .hex() call.

The proposed spec wording, as is, would mean that this should be an error, right? Because "member int of float | int has no method hex." builtins.int in typeshed doesn't have the hex method. So if you want to call hex you have to guard it with isinstance(float).

But this doesn't seem to be an error in either mypy or pyright. The latter is particularly interesting, since my understanding was that pyright already used the float | int interpretation of float annotations.

@erictraut What's the explanation of pyright's behavior here? Is this special-cased in some way?

I think if the conformance suite shows f.hex() here to not be an error, then that needs to be justified with some additional wording in the spec.

[erictraut]

In pyright's current implementation, I have a function called expandPromotionTypes that is responsible for taking the type float and expanding it to Float | int and complex into Complex | Float | int. (I use a capitalized name here to indicate the "real" type.)

There are three places where I initially added calls to this function:

1. isinstance type narrowing
2. Class pattern matching (which is the match statement equivalent of isinstance checks)
3. Attribute access ("dot") expression forms

I also 4) changed the inference logic for float literals to be inferred as Float rather than float.

I ended up backing out 3 and 4 because these two cases produced too much noise, including some false positive errors. See this issue, which shows some of the pain this caused pyright users.

I think it's reasonable to add 3 back, but doing so requires an additional change to avoid some of the noise. Namely, a call to float() or complex() constructors needs to evaluate to Float and Complex, respectively.

I think that adding 4 back would be problematic, especially in situations where float literals are used in list, set and dict expressions. These are problematic because the types are invariant. Consider the following code:

x = [3.1] # Should the type of `[3.1]` be inferred as `list[float]` or `list[Float]`?
x.append(1) # Should this be allowed? Most devs would expect it to be!

I think the expression [3.1] should continue to be inferred as list[float], which probably means that 3.1 should continue to be inferred as float and not Float. I think that's OK, but it does lead to an apparent inconsistency because [float(3.1)] is evaluated as list[Float]. The typing spec doesn't dictate type inference rules (currently), so these are not in scope for the spec, but this issue is something that type checker maintainers / authors will need to consider.

Here's a PR (and "mypy_primer" run) that adds back 3 from my list above. This makes pyright conformant with the proposed language in this typing spec update (i.e. it now generates an error for f.numerator in the conformance test case). The "mypy_primer" run shows one change in the dd_trace library. It's in code that looks like this:

x = {"a": float(0)} # Now evaluates to `dict[str, Float]` rather than `dict[str, float]`
x["b"] = 1 # Now a type error because `int` cannot be assigned to this `dict`

This change is not without consequences, but I think the impact is relatively minimal and new type errors are straightforward to fix.

[carljm]

I see, so pyright's implementation doesn't actually expand float to float | int when it's encountered in an annotation, but rather at certain points where the type is used.

I guess with #3 added back, pyright would also error on f: float; f.hex(). The hex/fromhex case is similar to the is_integer case encountered in microsoft/pyright#6032, but I expect is_integer is more commonly used. And is_integer was added to the int type in Python 3.12 to avoid this problem. It doesn't seem like your mypy primer run encountered any issues with calls to .hex or .fromhex, which doesn't surprise me.

One thing that does surprise me on that mypy primer run is that the two new errors don't appear to involve attribute access (your (3)), but rather inferred types for containers (your (4)). E.g. one of the errors is on this line: https://github.com/DataDog/dd-trace-py/blob/main/ddtrace/llmobs/_integrations/bedrock.py#L43

[erictraut]

so pyright's implementation doesn't actually expand float to float | int when it's encountered in an annotation

That's correct. It would be confusing for pyright & pylance users to see float | int in hover text, inlined type annotations, completion suggestions, reveal_type text, etc. if they use float in a type annotation. And likewise, if they specify float | int in a type annotation, it would be confusing to reduce that to just float. I try to retain the same form used in the annotation where possible.

[carljm]

Looks good to me.

[DiscordLiz]

IF this is actually going to be pushed through as a "Clarification", I'd appreciate if it was clear on what the behavior with TypeIs[float], and Type[T], where the parameter passed is float.

Both of these are runtime use of float, but are in an annotation context. TypeIs has a case where it either contradicts the guidance to treat TypeIs with the same narrowing rules as isinstance, or conflicts with the statement that float means float | int in an annotation context.

[DiscordLiz]

There's also a case that should have an explicit ruling with overload compatability, shown here: https://discuss.python.org/t/clarifying-the-float-int-complex-special-case/54018/44 as the changes do have other effects that break existing cases.

[JukkaL]

I'm still in the process of figuring out the implications to mypy (which uses a differerent approach currently), in case this would surface some additional edge cases worth documenting. I'm generally positive about clarifying the spec here though.

[JelleZijlstra]

Yes, this approach would imply changes to how mypy currently implements the special case. An alternative could be to specify mypy's behavior (which is essentially to pretend that int is a subclass of float), but I think that behavior is harder to understand and harder to specify precisely.

[beauxq]

It's not clear to me which of these should report errors, and which shouldn't.

def foo(x: float):
    assert_type(x, float | int)


def goo(x: float):
    assert_type(x, float)


def hoo(x: float | int):
    assert_type(x, float)

[JelleZijlstra]

This follows from the definitions in https://typing.python.org/en/latest/spec/annotations.html#type-and-annotation-expressions . Every time we're in a type expression, float means float | int.

[carljm]

(So to spell it out, the answer is that none of those should report an error, because "parameter annotation" and "second argument to assert_type" are both type expressions, so all of those examples are equivalent to each other, and every occurrence of | int is redundant.)

[llucax]

Any news on this? It looks like it got a few approvals but it didn't have any activity for a long time.

[JelleZijlstra]

I'm not sure we'll get consensus on this. Which is a shame since the current text in the spec seems unsatisfactory too.

[beauxq]

I think the best road forward is for type checkers to just start following what the spec currently says.

I recognize that there are downsides to that, but I don't see a better path than that, towards getting rid of the special case.

[carljm]

If we have consensus that "getting rid of the special case" is the best long-term approach, then the first necessary practical step is a re-annotation of typeshed to turn a lot of float argument annotations into float | int. This should be a no-op for type checkers that implement the special case (which is all of them, in some form), but is a pre-requisite to enable any type checker to even experiment with an option to disable the special case.

We have the same problem with all existing third-party library annotations in the world, and it's less clear to me how we can ever do a similar migration there.

I don't know if we have consensus on "remove the special case" as the ultimately desirable outcome. Personally I definitely wish the special case did not exist, but I'm not sure I see any realistic path to eliminating it.

[jorenham]

but I'm not sure I see any realistic path to eliminating it.

Maybe we could add a simple configuration option in py.typed that allows disabling these promotion rules on a per-project basis?

And then we might even be able to switch the default (i.e. disable int/float/complex promotions by default) after a long time (Python 3.20 or something). But that should probably depend on on how the community has adapted at that point.

[davidhalter]

but I'm not sure I see any realistic path to eliminating it.

Maybe we could add a simple configuration option in py.typed that allows disabling these promotion rules on a per-project basis?

And then we might even be able to switch the default (i.e. disable int/float/complex promotions by default) after a long time (Python 3.20 or something). But that should probably depend on on how the community has adapted at that point.

I think that's a pretty sensible solution to this problem. We might even want to introduce something like Rust editions where a specific year (e.g. the Python typing edition 2027) would break with some backwards compatibility of the older (default) edition. But that edition would need to be opt-in.

[srittau]

If we have consensus that "getting rid of the special case" is the best long-term approach, then the first necessary practical step is a re-annotation of typeshed to turn a lot of float argument annotations into float | int.

I actually have considered proposing this for typeshed. We could just replace float with float | int. This would introduce a small number of false positives negatives, but I wouldn't be too worried about that. Another alternative would be to introduce a temporary Float type alias to int | float that marks types that haven't been checked, yet. New code should not use that.

Overall I think we should start rather sooner than later with removing the special case, even if it will take a long time to remove.

[jorenham]

This would introduce a small number of false positives, but I wouldn't be too worried about that.

When I tried replacing all float with float | int and complex with complex | float in scipy-stubs a while back (scipy/scipy-stubs#492), Pyright became significantly slower. But that could very well have been specific to scipy-stubs, or something that the latest pyright version already resolved, so it's certainly worth trying out in typeshed.

Anyway, I guess my point is that performance hits are also something to be mindful of when doing this.

[carljm]

I actually have considered proposing this for typeshed. We could just replace float with float | int. This would introduce a small number of false positives, but I wouldn't be too worried about that.

Replacing parameter types should never result in false positives. Function return types should ideally be modified depending on the actual behavior of the function: can it return an int or a float, or can it actually return only a float? False positives could occur if we replace all return types without considering the actual function behavior.

Mutable attributes are the most difficult case, but if all type checkers allow setting them to an int already, then it's not really a false positive to start reporting that they could have an int value.

[srittau]

Replacing parameter types should never result in false positives.

Sorry, I meant false negatives.