Change Details

Some discussion in T9252 and T182. When some other application tries to acquire a lease on a Drydock resource (like Harbormaster performing a build, or Diffusion performing a merge), we need a mechanism to decide which blueprints the application may use. In particular, there are security/trust concerns in the T182 use case. We can't let merges and builds share the same resources, because builds are less-trusted and may compromise merges. In general, these properties should hold: - Merges must only acquire high-trust resources. - Builds must never acquire high-trust resources. But I think it goes further than this: I probably should not be able to compromise builds by adding a new blueprint that I claim is "high trust". The broader version of this is that it's surprising if you spin up a new tier for Android builds and the whole thing is instantly saturated by iPhone builds just because the hosts are technically compatible. This is bad on both sides: it's not what you wanted, and it's not what whoever owns the iPhone builds wanted. So it's bad if jobs use surprising blueprints, and it's bad if blueprints are used by surprising jobs. This makes me lean toward requiring explicit, opt-in authorization on both ends of the relationship: build plan X would explicitly say that it's OK to use blueprint Y, and blueprint Y would explicitly agree that it is usable by build plan X. I think this is good in terms of preventing abuse (and preventing surprises). The obvious drawback is that it is very inflexible. However, I think this selection mechanism may not need to be very flexible. Here are some use cases I can come up with: - You want to add or remove hosts from a pool. You can already do this through Almanac. With virtualized blueprints in the future, this is automatic. - You want to move a lot of build plans from static allocation (Almanac) to dynamic allocation (EC2). It seems OK/reasonable to do these somewhat individually so you can test them? You get a list of the plans you have to move anyway by seeing all the authorized plans. You probably shouldn't have hundreds of plans (parameterize a smaller number of plans instead?), and this doesn't seem terrible overall? - You want to write a build plan today and automatically add more host types later without re-authorizing (you do Linux builds today, but know you'll do Windows builds in the future -- or use only EC2 today, but know you'll use a mixture of EC2 and GCE in the future). We could write a ProxyBlueprint which just acts as a collection of other blueprints of the same type. - You want to change where merges happen for a lot of repositories, and you have like 300 repositories, and merge config is still per-repository. This one seems somewhat legitimate (more reasonable to have 300 repos than 300 build plans) and a bit annoying. Maybe merge config shouldn't be per-repostitory (e.g., default application-level merge config)? Maybe CLI tools? Maybe batch edit? Seems tractable and rare. Some of these are a little cumbersome but none seem unmanageably bad to me. @hach-que, are there cases I'm missing where double approval would pose an unreasonably high maintenance burden in your use cases? (This mechanism would not replace a future ability to say "give me a host with attributes X, Y, and Z", it would just control which blueprints are eligible to be used to satisfy the request.)

Some discussion in T9252 and T182. When some other application tries to acquire a lease on a Drydock resource (like Harbormaster performing a build, or Diffusion performing a merge), we need a mechanism to decide which blueprints the application may use. In particular, there are security/trust concerns in the T182 use case. We can't let merges and builds share the same resources, because builds are less-trusted and may compromise merges. In general, these properties should hold: - Merges must only acquire high-trust resources. - Builds must never acquire high-trust resources. But I think it goes further than this: I probably should not be able to compromise merges by adding a new blueprint that I claim is "high trust". The broader version of this is that it's surprising if you spin up a new tier for Android builds and the whole thing is instantly saturated by iPhone builds just because the hosts are technically compatible. This is bad on both sides: it's not what you wanted, and it's not what whoever owns the iPhone builds wanted. So it's bad if jobs use surprising blueprints, and it's bad if blueprints are used by surprising jobs. This makes me lean toward requiring explicit, opt-in authorization on both ends of the relationship: build plan X would explicitly say that it's OK to use blueprint Y, and blueprint Y would explicitly agree that it is usable by build plan X. I think this is good in terms of preventing abuse (and preventing surprises). The obvious drawback is that it is very inflexible. However, I think this selection mechanism may not need to be very flexible. Here are some use cases I can come up with: - You want to add or remove hosts from a pool. You can already do this through Almanac. With virtualized blueprints in the future, this is automatic. - You want to move a lot of build plans from static allocation (Almanac) to dynamic allocation (EC2). It seems OK/reasonable to do these somewhat individually so you can test them? You get a list of the plans you have to move anyway by seeing all the authorized plans. You probably shouldn't have hundreds of plans (parameterize a smaller number of plans instead?), and this doesn't seem terrible overall? - You want to write a build plan today and automatically add more host types later without re-authorizing (you do Linux builds today, but know you'll do Windows builds in the future -- or use only EC2 today, but know you'll use a mixture of EC2 and GCE in the future). We could write a ProxyBlueprint which just acts as a collection of other blueprints of the same type. - You want to change where merges happen for a lot of repositories, and you have like 300 repositories, and merge config is still per-repository. This one seems somewhat legitimate (more reasonable to have 300 repos than 300 build plans) and a bit annoying. Maybe merge config shouldn't be per-repostitory (e.g., default application-level merge config)? Maybe CLI tools? Maybe batch edit? Seems tractable and rare. Some of these are a little cumbersome but none seem unmanageably bad to me. @hach-que, are there cases I'm missing where double approval would pose an unreasonably high maintenance burden in your use cases? (This mechanism would not replace a future ability to say "give me a host with attributes X, Y, and Z", it would just control which blueprints are eligible to be used to satisfy the request.)

Some discussion in T9252 and T182. When some other application tries to acquire a lease on a Drydock resource (like Harbormaster performing a build, or Diffusion performing a merge), we need a mechanism to decide which blueprints the application may use. In particular, there are security/trust concerns in the T182 use case. We can't let merges and builds share the same resources, because builds are less-trusted and may compromise merges. In general, these properties should hold: - Merges must only acquire high-trust resources. - Builds must never acquire high-trust resources. But I think it goes further than this: I probably should not be able to compromise buildsmerges by adding a new blueprint that I claim is "high trust". The broader version of this is that it's surprising if you spin up a new tier for Android builds and the whole thing is instantly saturated by iPhone builds just because the hosts are technically compatible. This is bad on both sides: it's not what you wanted, and it's not what whoever owns the iPhone builds wanted. So it's bad if jobs use surprising blueprints, and it's bad if blueprints are used by surprising jobs. This makes me lean toward requiring explicit, opt-in authorization on both ends of the relationship: build plan X would explicitly say that it's OK to use blueprint Y, and blueprint Y would explicitly agree that it is usable by build plan X. I think this is good in terms of preventing abuse (and preventing surprises). The obvious drawback is that it is very inflexible. However, I think this selection mechanism may not need to be very flexible. Here are some use cases I can come up with: - You want to add or remove hosts from a pool. You can already do this through Almanac. With virtualized blueprints in the future, this is automatic. - You want to move a lot of build plans from static allocation (Almanac) to dynamic allocation (EC2). It seems OK/reasonable to do these somewhat individually so you can test them? You get a list of the plans you have to move anyway by seeing all the authorized plans. You probably shouldn't have hundreds of plans (parameterize a smaller number of plans instead?), and this doesn't seem terrible overall? - You want to write a build plan today and automatically add more host types later without re-authorizing (you do Linux builds today, but know you'll do Windows builds in the future -- or use only EC2 today, but know you'll use a mixture of EC2 and GCE in the future). We could write a ProxyBlueprint which just acts as a collection of other blueprints of the same type. - You want to change where merges happen for a lot of repositories, and you have like 300 repositories, and merge config is still per-repository. This one seems somewhat legitimate (more reasonable to have 300 repos than 300 build plans) and a bit annoying. Maybe merge config shouldn't be per-repostitory (e.g., default application-level merge config)? Maybe CLI tools? Maybe batch edit? Seems tractable and rare. Some of these are a little cumbersome but none seem unmanageably bad to me. @hach-que, are there cases I'm missing where double approval would pose an unreasonably high maintenance burden in your use cases? (This mechanism would not replace a future ability to say "give me a host with attributes X, Y, and Z", it would just control which blueprints are eligible to be used to satisfy the request.)