How can I interact with a tensor product with two different `by = var` variables?

Example:

library(mgcv)
N=1000
x1 = seq(1:N)
x2 = log(x1)
x3 = sqrt(x1)
fac1 = ceiling(rnorm(N)*3)
fac2 = ceiling(runif(N)*3)
y = fac1*x2 + x1*x2 + x2 + x3*x2 + x2*(x1/x3)^(.8+fac2/10) + rnorm(N)*x2

mod = gam(y~
        s(as.factor(fac1),bs="re",by=x2) 
    +       s(x2)
    +   s(x1,by=x2)
    +   s(x3,by=x2)
    +   te(x1,x3,by=x2, by=as.factor(fac2))
    )   

The last tensor does not want to let me interact with him twice. The first is a continuous variable that multiplies each term in the model matrix, and the second is a factor - it creates a different surface for each level of the factor.

EDIT: The last member will be equivalent te(x1*x2,x1*x3,by=as.factor(fac2)). But if I come up with a new variable x1x2 = x1*x2, I will lose the ability to callpredict.gam

How do I program this? Do I need to refer to SmoothConor to such? If so, then a use case would be really helpful.

Thank!

(PS: I know about the heteroskedastic nature of the model. Feature, not error.)